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PREFACE 


This  report  describes  a  portion  of  a  study  of  Air  Force  aircrew 
training  using  simulation  as  one  part  of  a  total  training  system.  The 
study  was  initiated  in  response  to  a  Request  for  Personnel  Research 
(RPR-77-9)  from  Headquarters,  USAF  (AF/XOOTD). 

This  is  one  of  seven  technical  reports  prepared  for  the  Air  Force 
Human  Resources  Laboratory,  Logistics  and  Technical  Training  Division, 
under  Contract  F33615-77-C-0067,  Simulator  Training  Requirements  and 
Effectiveness  Study  (STRES).  The  reports  are  identified  in  Chapter  II 
of  this  document. 

The  work  was  performed  from  August  1977  through  February  1980  by  a 
team  made  up  of  Canyon  Research  Group,  Inc.;  Seville  Research 
Corporation;  and  United  Airlines  Flight  Training  Center.  Canyon^ 
^Jtesearch  Group^  Inc.  was  the  prime  contractor;  Mr.  Clarence  A.  Semple 
was  the  Program~Maha'g€Fr;;  The  Seville  Research  Corporation  effort  was 
headed  by  Dr.  Paul  U.  Cart).  The  United  Airlines  effort  was  headed 
initially  by  Mr.  Dale  L.  Seay  and  subsequently  by  Mr.  Kenneth  E.  Allbee. 

Mr.  Bertram  W.  Cream  was  the  AFHRL/AS  Program  Manager.  Other  key 
members  of  the  AFHRL/AS  technical  team  included  Dr.  Thomas  Eggemeier  and 
Dr.  Gary  Klein.  A  tri -service  STRES  Advisory  Team  participated  in 
guid’ng  and  monitoring  the  work  performed  during  this  contract  to  assure 
its  operational  relevance  and  utility.  Organizational  members  of  the 
Advisory  Team  were: 

Headquarters,  USAF 

Headquarters,  Air  Training  Command 

Headquarters,  Tactical  Air  Comnand 

Headquarters,  Strategic  Air  Command 

Headquarters,  Military  Airlift  Comnand 

Headquarters,  Aerospace  Defense  Command 

Headquarters,  Air  Force  Systems  Comnand 

Tactical  Air  Command,  Tactical  Air  Warfare  Center 

Air  Force  Human  Resources  Laboratory 

USAF  Aeronautical  Systems  Division 

Air  Force  Test  and  Evaluation  Center 

Air  Force  Manpower  and  Personnel  Center 

Air  Force  Office  of  Scientific  Research 

Navy  Training  Analysis  and  Evaluation  Group 

Army  Research  Institute  for  the  Behavioral  and  Social  Sciences 

The  authors  wish  to  express  their  gratitude  to  the  hundreds  of 
people  In  the  United  States  Air  Force,  Navy,  Army,  Coast  Guard,  NASA, 
FAA  and  Industry  who  contributed  to  this  program  by  providing  technical 
data  and  participating  in  Interviews  and  technical  discussions  during 
program  data  collection. 
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CHAPTER  I 


INTRODUCTION 

The  U.S.  military  services  have  been  users  of  flight  training 
devices  and  simulators  for  over  half  a  century.  These  training  media, 
known  collectively  as  aircrew  training  devices  (ATD),  include  cockpit 
familiarization  and  procedures  trainers,  part-task  trainers,  operational 
flight  trainers,  weapons  systems  trainers,  and  full  mission  trainers. 
In  recent  years,  use  of  ATDs  has  increased  to  the  point  that  the  devices 
represent  major  aircrew  training  resources,  and  their  effective  and 
efficient  design  and  use  is  a  matter  of  continuing  concern. 

In  response  to  this  concern,  the  U.S.  Air  Force  undertook  a 

programmatic  study  of  factors  involved  In  their  design  and  use.  This 
program  was  titled  Simulator  Training  Requirements  and  Effectiveness 
Study  (STRES).  The  general  objectives  of  STRES  are  to  define,  describe, 
collect,  analyze  and  document  information  bearing  on  the  cost  and 
training  effectiveness  of  flight  simulators.  Topic  areas  covered  in  the 
program  are:  fidelity;  instructional  support  features;  simulator 
utilization;  life  cycle  costs;  and  worth  of  ownership.  Products  of  the 
program  are  intended  for  use  by  those  who  manage  and  use  simulators  for 
training,  evaluate  simulator  requirements,  design,  procure,  and  maintain 
these  devices.  Chapter  II  describes  the  STRES  effort  in  more  detail. 

This  volume  is  one  of  seven  prepared  during  STRES.  It  addresses 
concerns  related  to  the  design  and  use  of  ATD  instructional  support 

features.  Other  volumes  prepared  during  the  program  are  identified  in 

Chapter  II. 

BACKGROUND 

The  history  of  flight  simulation  has  been  one  of  constant 

technological  improvements.  Most  of  these  have  focused  on  Improving 
fidelity.  As  a  result,  modern  digital  flight  simulators  look,  feel  and 
respond  more  like  their  aircraft  counterparts  than  ever  before.  One 
effect  has  been  Improved  acceptance  of  simulators  by  Instructors  and 
students . 

An  effectively  designed  training  simulator,  however,  is  one  that  not 
only  promotes  user  acceptance,  but  also  takes  advantage  of  the  unique 
training  opportunities  that  can  be  provided  through  simulation.  ATDs 
offer  freedom  from  many  of  the  instructional  constraints  associated  with 
aircraft  as  training  devices.  For  example.  Instructional  efficiencies 
can  easily  be  obtained;  performance  assessment  opportunities  are 
Improved;  and,  new  tactics  can  be  evaluated,  which  might  not  be  possible 
in  the  air. 
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PURPOSE  OF  THIS  REPORT 


The  purpose  of  this  report  is  to  define  a  comprehensive  set  of 
instructional  support  features,  develop  criteria  for  matching  the 
features  with  training  requirements,  and  present  principles  for  their 
design  and  use.  Specifically,  the  objective  of  this  report  is  to 
evaluate  currently  installed  features  and  review  newly  evolving 
technology  for  future  applications. 

This  report  is  based  on  analysis  of  currently  installed 
instructional  features  and  a  review  of  the  scientific  and  professional 
literature.  Gaps  exist  in  this  evolving  technological  area.  The  gaps 
remain  to  be  filled  by  experimental  research.  Thus,  the  report  also 
identifies  information  shortfalls  on  the  design  and  use  of  instructional 
support  features  to  alert  users  of  the  report  to  potential  risk  areas, 
as  well  as  to  guide  future  research. 

It  is  anticipated  that  primary  users  of  this  report  will  be 
operational  training,  management  and  engineering  personnel  tasked  with 
developing  ATD  specifications.  Additional  users  may  include  engineers 
and  training  psychologists  involved  in  detailed  ATD  design;  operational 
users  whose  job  Is  aircrew  training  using  ATDs;  and  operational 
personnel  and  training  psychologists  tasked  with  test  and  evaluation  of 
instructional  support  features. 

FIDELITY-PLUS 

For  years,  training  psychologists  have  emphasized  that  ultra  high 
fidelity  probably  Is  not  necessary  for  training  many  aircrew  tasks. 
They  have  further  argued  that  training  effectiveness,  and  particularly 
training  efficiency,  could  be  improved  if  ATD  designs  did  not  mimic 
corresponding  aircraft  and  systems  so  closely.  Their  points  are  valid, 
but  their  approach  has,  on  occasion,  lead  to  confusion. 

One  theme  of  their  argunent  has  been  that  designing  ATDs  to  be 
capable  of  doing  only  those  things  possible  in  aircraft  may  result  in 
high  fidelity,  but  does  not  take  advantage  of  the  fact  that  ATDs  are  not 
aircraft.  A  second  theme  has  been  that  adding  capabilities  to  ATDs  that 
are  not  Inherent  in  aircraft  results  In  lower  fidelity.  The  baseline 
for  the  second  theme  Is  that  whatever  the  aircraft  can  do  equals  100% 
fidelity,  and  any  departure  from  this,  either  in  terms  of  lesser 
capabilities  or  added  capabilities,  Is  a  departure  from  100%  fidelity. 
This  approach  has  seriously  confused  both  the  fidelity  issue  and  the  ATD 
Instructional  design  Issue. 

Fidelity-plus  is  way  of  thinking  about  how  to  design  and  use  ATDs  as 
instructional  tools  while  avoiding  the  nagging  apprehension  that 
fidelity  must  be  sacrificed  In  the  process.  For  example,  highly  accurate 
flight  characteristics  fidelity  does  not  need  to  be  sacrificed  if  the 
capability  to  freeze  one  or  more  axes  of  flight  control  is  built  into 


the  ATD.  Content  of  an  out  of  the  cockpit  visual  scene  does  not  need  to 
be  degraded  so  that  the  ability  to  add  cues  of  benefit  to  the  student's 
learning  process  can  be  done  at  the  instructor's  discretion.  Also,  the 
capability  to  use  an  ATD  for  chock  to  chock  mission  training  is  not  lost 
just  because  the  device  also  incorporates  capabilities  allowing  the 
instructor  to  freeze  the  mission  temporarily,  or  to  reinitialize  the  ATD 
to  some  other  geographic  point  at  the  press  of  a  button. 

In  short,  fidelity  does  not  need  to  be  sacrificed  to  design  and  use 
simulators  as  unique  instructional  tools  that  are  not  constrained  by  the 
ways  aircraft  operate  and  must  be  operated. 

INSTRUCTIONAL  SUPPORT  FEATURES 

It  is  important  to  distinguish  between  fidelity  features  that  enable 
certain  training  to  be  undertaken  in  ATDs,  and  instructional  support 
features.  An  out  of  the  cockpit  visual  scene  is  a  fidelity  feature  that 
can  make  possible  visual  landing,  air  to  ground  weapon  delivery  or  air 
refueling  training.  Such  visual  task  training  would  not  be  possible 
without  the  fidelity  feature.  However,  some  have  referred  to  out  of  the 
cockpit  visual  simulation  as  an  "instructional  feature"  because  it  makes 
certain  instruction  possible.  This  is  not  how  the  term  instructional 
support  feature  is  used  in  this  report.  As  used  here,  instructional 
support  features  are  ATD  hardware  and  software  capabilities  that  allow 
the  instructor  to  manipulate,  supplement  and  otherwise  control  the 
student's  learning  experiences  with  the  Intent  of  promoting  the  rate  at 
which  skills  are  learned  and  maximizing  the  levels  of  skills  achieved 
(Hughes,  1978). 

There  has  been  a  long  term  interest  in  finding  ways  to  capitalize  on 
unique  training  opportunities  offered  by  properly  designed  ATDs  (Smode 
and  Hall,  1966;  USAF,  1978;  and  Hughes,  1978).  A  recent  survey  (Caro, 
1977)  found  that  unique  ATD  training  capabilities  often  are  not 
capitalized  on.  The  same  was  found  true  in  the  present  program. 

Instructional  support  features  offer  one  means  of  capitalizing  on 
unique  training  capabilities  inherent  in  well  designed  and  used  ATDs. 
Virtually  all  ATDs  incorporate  some  type  of  instructional  support 
feature  intended  to  promote  training  quality  control,  efficiency  and 
effectiveness.  Some  features,  such  as  record/replay  and  problem  freeze, 
have  been  Incorporated  into  ATDs  for  a  number  of  years.  Others,  such  as 
programed  mission  event  control  and  automated  performance  measurement, 
are  relatively  new  technologies.  Still  others,  such  as  automated 
adapative  training,  are  largely  although  not  totally  in  laboratory 
stages  of  development. 

Instructional  support  features  offer  many  potential  ATD  training 
benefits  when  they  are  properly  designed  and  used.  These  Include: 


13 


Making  it  possible  to  use  productive  training 
methods  that  cannot  be  used  in  aircraft; 

Tailoring  training  to  individual  student  needs; 

Making  more  productive  use  of  available  training 
time  and  instructor  assets; 

Enhancing  the  standardization  of  training  and  the 
evaluation  of  operational  readiness; 

Relieving  instructors  of  routine,  non-productive 
tasks  through  automation; 

Controlling  the  simulated  environment  and  aircraft 
conditions; 

Improving  capabilities  to  diagnose  student  learning 
problems  and  focus  instruction; 

Providing  learning  oriented  performance  feedback; 
and 

Controlling  training  problem  complexity  and 
difficulty. 

INSTRUCTIONAL  SUPPORT  FEATURES  AND  TRAINING  TASKS 

Many  designers  and  users  of  ATD  instructional  support  features  have 
assumed  that  the  value  of  a  feature  is  directly  linked  to  the  type  of 
task  being  trained.  They  have  assumed,  for  example,  that  some  features 
are  best  suited  for  training  approach  and  landing,  while  other  features 
are  better  suited  for  training  air  refueling. 

This  assumption  was  evaluated  throughout  the  program  and  was  found 
to  be  incorrect.  What  was  found  is  that  the  value  of  instructional 
features  hinges  strongly  on  the  student's  stage  of  training  (i.e.  skill 
and  knowledge  levels)  and  the  ways  the  feature  can  be  used  by 
Instructors  to  support  the  training  of  students  at  different  stages  of 
skill  development.  This  finding  had  an  important  Impact  on  the 
organization  and  content  of  this  report,  as  discussed  in  the  next 
section. 

INSTRUCTOR  FUNCTIONS  AND  INSTRUCTIONAL  SUPPORT  FEATURES 

An  Important  thrust  of  this  report  is  that  the  design  or  use  of  any 
instructional  support  feature  has  meaning  only  In  the  context  of: 

1)  the  manner  in  which  It  supports  the  instructor  in  providing 
meaningful  Instruction  effectively  and  efficiently;  and  2)  the  manner  jn 
which  it  supports  the  student  by  helping  him  to  learn  faster  and  perform 
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better.  In  other  words.  Instructional  support  feature  design  and  use 
roust  focus  on  the  payoffs  they  can  provide.  Hardware  and  software 
characteristics  of  any  feature,  by  themselves,  have  little  practical 
meaning. 

Organization  of  this  report  focuses  on  the  instructor  and  student 
support  roles  discussed  above.  The  primary  organizing  mechanism  was  to 
ask:  Can  a  particular  instructional  support  feature  be  a  useful  tool 
for  instructors  in  using  ATDs  in  the  doing  of  their  job?  An  answer  to 
this  question,  in  turn,  requires  some  analysis  of  what  an  instructor 
does  during  training  in  ATDs. 

The  instructor's  job  is  made  up  of  many  activities  that  contribute 
to  the  larger  role  of  training.  Functions  are  used  here  to  describe  the 
general  types  of  activities  that  instructors  (and,  occasionally,  console 
operators)  perform  in  the  use  of  simulators  for  training. 

Functions  performed  by  people  often  are  more  difficult  to  define  and 
get  agreement  on  than  are  functions  performed,  for  example,  by  major 
aircraft  subsystems.  This  is  true  of  instructor  functions.  The  main 
reasons  is  that  the  process  of  instruction  often  involves  doing  several 
functions  at  the  same  time  or  in  relation  to  one  another.  In  other 
words,  it  is  acknowledged  that  the  instructor's  job  is  not  simply  a 
series  of  duties  performed  in  isolation  from  each  other.  For  example, 
individualizing  training  Is  not  done  in  isolation  from  monitoring  and 
evaluating  performance.  The  two  go  together. 

fti  the  other  hand,  if  it  is  acknowledged  that  instructor  functions 
are  not  really  independent,  then  the  various  functions  can  at  least  be 
dealt  with  one  at  a  time  in  order  to  define  them,  examine  relationships 
among  them,  and  to  illustrate  how  instructional  feature  technology  may 
help  instructors  in  doing  these  many  things. 

Table  1  defines  seven  instructor  functions  involved  in  the  use  of 
ATDs.  Instructional  support  feature  technology  that  has  known  or 
potential  value  as  a  tool  for  the  instructor  to  use  in  performing  eacir 
function  also  is  presented.  The  functions  and  their  definitions  are 
adapted  from  previous  work  (Semple,  Vruels,  Cotton,  Durfee,  Hooks,  A 
Butler,  1979)  that  dealt  with  the  Instructor's  role  in  pilot  training 
and  the  development  of  Instructional  support  technology  to  aid  the 
instructor  in  performing  his  role. 

The  functions  presented  In  this  section  are  limited  to  those 
Involved  in  the  use  of  ATDs.  Thus,  they  do  not  address  other  job 
requirements  such  as  collateral  duties,  platform  lecturing,  or  the 
development  of  instructional  materials.  Also,  it  Is  recognized  that 
details  of  the  way  each  function  Is  performed  In  the  operational 
environment  vary  depending  on  student  characteristics,  the  type  of 
device  being  used  and  the  types  of  training  provided.  Effects  of  such 
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Table  1.  Instructor  Functions  and  Related  ATD 
Instructional  Support  Features 


Instruct.  Instruct  means  to  direct  the  growth  of  skills  and  knowledges 
by  giving  knowledge  and  information  in  a  systematic  way.  Related 
support  features  are: 

Freeze;  automated  demonstrations;  record  and  replay; 
manual  and  prograirmable  sets  of  initial  conditions; 
automated  cuing  and  coaching;  ATD-mounted  audio/visual 
media;  and  manual  or  programmable  malfunction  control. 

Moni tor/E val uate  Performance .  Monitor  means  to  observe  and  check  the 
quality  of  performance  Tor  specific  purposes.  Evaluate  means  to 
determine  the  meaning  and  significance  of  observed  performance  values. 
Related  support  features  are: 

Automated  performance  measurement;  automated 
performance  alerts;  closed  circuit  TV;  and  performance 
monitoring  displays  (plotter/position  displays, 
perspective  displays,  event  annunciators,  repeater 
Instruments,  and  synthesized  (CRT)  repeater 
Instrunents. 

Control /Individualize  Training.  Control  means  to  direct  or  regulate 
training  events.  Individualizing  training  Is  a  special  case  of 
controlling  where  training  Is  tailored  to  the  student's  specific  skills 
and  abilities.  Related  support  features  are: 

Automated  adaptive  training;  and  programned  mission 
scenarios 

Controller.  Instructors  or  ATD  operators  often  play  the  roles  of  air 
traffic  controllers,  or  control  the  actions  of  simulated  threats. 
Related  support  features  are: 

Automated  controllers  (Incorporating  computer 
generated  voice  and  speech  understanding  systems); 
graphic  or  text  readouts  of  controller  Information; 
and  computer  controlled  (Iron  pilot)  adversaries. 
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Table  1.  (Continued) 


Prepare.  Prepare  means  working  out  beforehand  the  details  of  what  is  to 
be  trained  and  how  it  is  to  be  trained  during  a  period  of  instruction. 
For  the  purposes  of  this  report,  preparing  focuses  on  highly 
Individualized,  proficiency  advanced  training  where  the  instructor  is 
required  to  make  training  content  decisions  as  a  matter  of  routine.  The 
related  support  feature  is: 

Computer  managed  insructional  system,  with  specific 
capability  to  track  training  objectives  achieved  and 
yet  to  be  achieved  separately  for  each  trainee. 

Brief.  Brief  means  giving  final  training  session  Instructions  to  the 
student ,  coaching  him  through  the  events  of  the  mission  in  advance, 
questioning  him  to  determine  his  readiness  for  the  upcoming  training, 
and  providing  remedial  instruction  in  weak  areas.  The  related  support 
feature  is: 


Recorded  briefings  of  training  session  instructions 
and  coaching  information. 

Debrief.  Debrief  means  to  review  with  the  goal  of  evaluation  combined 
wTtFTTnstruction.  Related  support  features  are: 


Remote  replay  of  flight  profile  or  crewstation  display 
Information;  and  hardcopy  printouts  of  flight  profile 
or  performance  measurement  data 


factors  are  noted,  when  appropriate,  in  the  information  presented  on 
each  instructional  feature. 


Not  all  relevant  issues  fall  conveniently  into  any  of  the  instructor 
function  categories.  The  following  two  issues  also  are  addressed  in 
this  report,  although  they  do  not  appear  in  Table  1:  1) 

instructor/operator  console  design/location;  2)  and  instructor/operator 
training  as  it  relates  to  the  use  of  ATD  instructional  support  features. 

REPORT  ORGANIZATION 

Separate  chapters  deal  with  each  instructor  function.  Content  of 
each  chapter  is  structured  to  provide  coirmon  sets  of  information  on  each 
related  instructional  support  feature.  The  following  information  is 
presented  in  the  main  body  of  this  report  separately  for  each 
instructional  support  feature. 


Summary.  The  discussion  of  each  instructional  support  feature 
starts  with  a  summary  statement-.  The  feature  is  defined,  and  the 
state  of  knowledge  on  the  design  and  use  of  the  feature  is 
sumnarized. 

Feature  Description.  Each  instructional  support  feature  directly 
related  to  a  specific  instructional  function  Is  described 
separately.  Where  a  particular  feature  supports  more  than  one 
instructor  function,  it  is  considered  in  detail  under  the  function 
to  which  It  has  the  strongest  relationship.  Each  feature  is  cross 
referenced  to  the  other  Instructor  functions  it  supports. 

Related  Support  Features.  Instructional  functions  are  interrelated. 
It  is  logical,  therefore,  that  instructional  support  features  should 
be  addressed  in  an  Integrated,  mutually  supportive  framework.  For 
example,  total  system  freeze  may  have  value  during  automated 
demonstrations.  Also,  automated  adaptive  training  is  predicated  on 
workable  automated  performance  measurement  and  automated  testing  for 
learning  problem  diagnosis.  This  section  presents  linkages  between 
the  feature  under  discussion  and  other  features,  as  appropriate. 

Instructional  Values.  The  Instructional  values  most  often  assumed 
or  attributed  to  each  feature  are  identified  and  discussed. 
Additionally,  this  section  presents  selected  Instructional 
principles  associated  with  the  proper  use  of  each  feature. 

The  U.S.  Army  (Wheaton,  Flngerman,  Rose  and  Leonard,  1976)  recently 
completed  a  program  to  Identify  principles  of  Instruction,  assess 
their  theoretical  benefits,  and  assess  research  evidence  on  the 
benefits  of  each  principle.  A  product  of  their  research  was  a 
theoretical  and  empirical  assessment  of  the  extent  to  which  support 
presently  exists  for  each  instructional  principle. 
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A  total  of  97  instructional  principles  was  assessed  by  Wheaton,  et 
al.  Each  was  rated  separately  on  the  basis  of  theoretical  and 
experimental  support.  Instructional  principles  of  apparent 
intuitive  value  are  relatively  easy  to  create.  The  question  is: 
Can  they  be  supported  on  sound  theoretical  and  empirical  bases? 
Often,  this  is  not  the  case.  Therefore,  for  this  report, 
instructional  principles  supported  by  an  empirical  confirmation 
rating  of  good  to  excellent  were  separated  from  the  rest  so  that  well 
established  instructional  principles  could  be  associated  with 
observed  and  projected  uses  of  the  instructional  support  features 
addressed  in  this  report. 

The  principles  thus  identified  are  presented  in  relation  to  commonly 
assumed,  often  untried,  but  potential  training  values  of 
instructional  support  features.  The  reasons  for  doing  so  are  to 
stimulate  creative  thought  on  applications  of  this  evolving 
technology  area  while  at  the  same  time  advising  on  realistic  bounds 
that  are  In  keeping  with  the  present  state  of  knowledge. 

Not  all  instructional  support  features  are  directly  related  to 
principles  of  training.  Some,  for  example,  simply  are  associated 
with  training  efficiency,  instructor  workload,  and  the  best  use  of 
training  time.  An  example  is  ATD  initialization  to  a  set  of 
conditions.  In  the  example,  time  required  to  "fly"  and  configure 
the  ATO  to  desired  states  can  be  shortcut.  In  other  Instances, 
training  principles  are  Involved  and  are  Identified. 

Observed  Applications.  Training  programs  and  devices  observed 
during  the  program  that  Incorporate  the  Instructional  support 
features  under  consideration  are  sunmarlzed.  The  purpose  is  to 
identify  existing  applications  of  the  feature  and  provide  the  reader 
with  a  general  background  on  Interview  and  observational  data 
sources. 

Obviously,  present  practices  In  the  design  and  use  of  Instructional 
support  features  should  not  set  limits  for  the  future.  Thus, 
present  day  designs  and  uses  are  discussed  only  when  they  provide 
illustrations  of  particularly  good  or  poor  characteristics  and, 
therefore,  may  be  of  value  to  the  reader. 

Utility  Information.  Content  of  this  section  deals  with  a  spectrum 
of  factors  that  can  Influence  acceptance  and  use  of  an  instructional 
feature.  The  range  of  factors  initially  investigated  Included: 
reliability  and  maintainability;  ease  of  use;  frequency  of  use;  user 
acceptance;  and  a  variety  of  variables  that  Impact  on  the 
acceptability  and  utility  of  Individual  Instructional  support 
features  and  combinations  of  features. 
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During  the  program  it  was  found  that  reliability  and  maintainability 
were  not  isolated  to  particular  features.  Rather,  these  factors  are 
embedded  within  total  ATD  reliability  and  maintainability.  For 
example,  automated  performance  measurement,  where  it  exists,  usually 
Is  not  a  separate  hardware/software  capability.  Rather,  it  is  an 
integral  part  of  the  ATDs  design.  Thus,  features  typically  do  not 
fail  in  isolation.  The  entire  simulation  either  operates  or  does 
not  operate. 

Similarly,  frequency  of  use  of  a  feature  is  more  a  function  of 
instructor/operator  training,  training  program  objectives  and 
student  skill  levels  than  anything  else.  For  example,  the  use  of 
record/replay  may  be  greater  when  the  student  is  relatively  naive  to 
the  task  than  when  he  is  highly  experienced.  This  section 
addresses,  feature  by  feature,  these  and  related  considerations. 

Related  Research.  Specific  research  results  and  other  information 
developed  during  the  program  are  presented  separately  for  each 
feature.  Information  shortfalls  also  are  set  forth  in  this  and 
related  sections.  On-going  research  activities  are  identified  as 
they  relate  to  present  Information  shortfalls. 

Instructional  support  features  represent  a  largely  neglected 
research  area.  Their  possible  values  and  limitations  are  largely, 
although  not  totally,  assumed  based  on  experience  and  judgement. 
This  report  combines  judgement,  experience  and  research  findings 
(where  they  exist)  regarding  the  values  and  limitations  of 
instructional  features  In  the  context  of  specific  Instructional 
functions  and  training  requirements.  Additional  research  on 
instructional  feature  design  and  use  is  sorely  needed.  Until  It  is 
performed,  many  design  and  use  recommendations  really  are  just 
hypotheses . 

Design  Considerations.  This  section  presents  guidelines  for 
implementing  Instructional  support  features  in  ways  that  will 
enhance  support  of  the  Instructional  process  and  promote  ease  and 
effectiveness  of  use  of  the  features  by  instructors,  operators  or 
students,  as  appropriate. 

Conditional  judgements  also  are  made  on  possible  designs  and 
applications  of  features  currently  In  developmental  stages  and  for 
which  no  operational  or  research  experience  of  consequence  exists. 
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CHAPTER  II 


THE  STRES  PROGRAM 


INTRODUCTION 

Aircrew  training  Is  an  expensive  and  time  consuming  endeavor.  At 
one  time  or  another,  virtually  every  known  training  method  and  medium 
has  been  used  to  develop  operationally  ready  aircrews  and  to  maintain 
their  skill  levels.  To  meet  these  training  needs  in  a  cost  effective 
manner,  the  U.S.  Military  has  shown  increased  interest  in  the  use  of 
simulators  and  related  training  devices.  These  training  media,  known 
collectively  as  aircrew  training  devices  (ATD),  include  cockpit 
familiarization  and  procedures  trainers,  part-task  trainers,  operational 
flight  trainers,  weapon  systems  trainers,  and  full  mission  trainers. 

Recent  requirements  to  economize  on  aircraft  fuel  used  for  training 
have  provided  strong  Impetus  for  the  Increased  Interest  in  ATDs,  but 
other  factors  have  contributed  as  well.  These  other  factors  include 
increasingly  congested  airspace,  safety  during  training,  cost  of 
operational  equipment  used  for  training,  and  a  desire  to  capitalize  on 
training  opportunities  that  ATOs  provide  for  training  that  cannot  be 
undertaken  effectively,  safely  or  economically  In  the  air. 

Because  of  the  advantages  simulation  can  offer  over  other  aircrew 
training  media.  It  Is  current  Air  Force  policy  that  ATDs  will  be  used  to 
the  fullest  extent  to  Improve  readiness,  operational  capability  and 
training  efficiency.  Implementation  of  this  policy  requires  specific 
technical  guidance.  Information  upon  which  to  base  that  guidance  is 
sparse,  however,  and  the  Information  that  does  exist  is  not  always 
available  to  those  who  need  It.  The  STRES  program  was  Intended  as  a 
means  of  identifying  and  making  available  ATD  design,  use,  cost  and 
worth  Information  to  support  relevant  Air  Force  policies.  The 
Information  Is  Intended  to  provide  guidance  for  the  enhancement  of 
present  training,  as  well  as  for  the  focus  of  research  and  development 
needed  to  enhance  future  simulation-based  training. 

PROGRAM  STRUCTURE 

The  primary  objectives  of  the  overall  STRES  program  are  to  define, 
describe,  collect,  analyze  and  document  information  bearing  on  four  key 
areas.  The  areas  are: 

Criteria  for  matching  training  requirements  with 
ATD  fidelity  features; 

Criteria  for  matching  ATD  instructional  features 
with  specific  training  requirements; 
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Principles  of  effective  and  efficient  utilization 
of  ATOs  to  accomplish  specific  training 
requirements;  and 

Models  of  factors  influencing  the  life  cycle  cost 
and  worth  of  ownership  of  ATDs. 

The  Air  Force  plan  for  accomplishing  these  objectives  involves  a 
four-phase  effort.  Phase  I,  which  was  concluded  prior  to  the  start  of 
the  present  studyt  was  an  Air  Force  planning  activity  to  define  and 
prioritize  the  total  effort.  Phase  II,  the  effort  described  in  the 
series  of  reports  identified  below,  was  a  29  month  study  that  Involved 
collecting,  integrating,  and  presenting  currently  available  scientific, 
technical,  and  operational  information  applicable  to  specific  aircrew 
training  issues.  Phase  II  also  Involved  the  identification  of  research 
and  development  efforts  needed  to  enhance  future  simulator  training. 
Phase  II  was  conducted  by  a  team  composed  of  Canyon  Research  Group, 
Inc.,  Seville  Research  Corporation,  and  United  Airlines  Flight  Training 
Center.  Phase  III  is  planned  to  be  a  research  activity  that  will 
provide  additional  information  on  important  simulation  and  simulator 
training  questions  that  cannot  be  answered  with  currently  available 
data.  Finally,  building  on  Phases  II  and  III,  Phase  IV  is  planned  as  an 
Air  Force  effort  to  Integrate  findings,  publish  relevant  information, 
and  provide  for  updating  of  the  knowledge  base  as  new  information 
becomes  available. 

A  tri-service  Advisory  Team  was  formed  by  the  Air  Force  to  help 
guide  STRES.  The  team  has  participated  in  two  ways.  One  was  to  assist 
in  the  Phase  I  program  planning.  The  second  has  been  to  provide 
guidance  and  evaluative  feedback  during  Phase  II  to  ensure  that  products 
of  the  phase  would  be  operationally  relevant  and  useful.  Operational 
users  of  ATDs  and  the  research  community  were  represented  on  the 
Advisory  Team. 

A  principal  task  of  the  Advisory  Team  was  to  participate  in  the 
development  of  objectives  and  guidelines  for  the  conduct  of  the  Phase  II 
technical  effort.  As  a  focus  for  those  efforts,  a  set  of  "high  value" 
operational  tasks  was  Identified.  The  tasks  selected  were  those  for 
which  potential  ATD  training  benefits  were  judged  to  be  greatest,  and 
for  which  Information  on  ATD  design,  retrofit,  use,  and  worth  was 
believed  to  be  Incomplete  or  lacking.  These  tasks  also  provided  a  focus 
for  identifying  questions  and  Issues  reflecting  the  information  needs  of 
operational  personnel  that  were  to  be  addressed  during  Phase  II  efforts. 
The  high  value  tasks  identified  by  the  Advisory  Team  are; 

Individual  and  formation  takeoff  and  landing; 

Close  formation  flight  and  trail  formation,  both 
close  and  extended; 
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Aerobatics; 

Spin,  stall  and  unusual  attitude  recognition, 
prevention  and  recovery; 

Low  level  terrain  following  flight; 

Air  refueling; 

Air-to-air  combat  (guns  and  missiles);  and 
Air-to-ground  weapons  delivery. 

SOURCES  OF  INFORMATION 

Information  from  two  sources  was  collected  during  Phase  II  to 
address  the  objectives  of  STRES.  One  source  was  the  professional  and 
technical  literature.  This  literature  included  books,  conference 
proceedings,  professional  journals,  research  reports,  military  manuals 
and  regulations,  and  policy  statements.  The  second  source  was  military 
and  civilian  personnel  whose  experiences  related  to  the  objectives  of 
the  study.  Information  was  obtained  from  these  personnel  during  visits 
to  organizations  to  which  they  were  assigned. 

Literature  Review 


Computer  searches  were  made  at  the  outset  to  identify  literature 
relevant  to  all  facets  of  the  Phase  II  effort.  In  addition,  each 
contractor  team  member  was  responsible  for  identifying  documents 
pertinent  to  his  responsibilities  that  may  have  been  missed  in  the 
computer  searches.  In  these  individual  efforts,  articles  pertinent  to 
the  various  activities  of  colleagues  were  regularly  encountered.  Each 
investigator  was  aware  of  the  information  needs  of  his  colleagues,  and 
frequent  communication  among  team  members  assured  that  colleagues  would 
be  apprised  of  articles  of  potential  value  to  their  tasks.  Hence,  the 
search  for  literature  of  concern  to  the  preparation  of  a  given  volume  of 
the  STRES  report  series,  while  systematically  compiled  by  those 
specifically  responsible  for  that  volume,  was  expanded  through  the 
efforts  of  the  entire  team. 

To  provide  Integration  and  focus  to  these  literature  search  efforts, 
one  group  of  the  STRES  team  was  specifically  responsible  for  identifying 
articles  of  potential  Interest  to  all  team  efforts,  as  well  as  for 
preparing  comprehensive  abstracts  of  selected  documents  that  appeared 
particularly  valuable.  These  abstracts  appear  in  a  separate  volume  of 
the  STRES  report  series. 

More  than  1,100  documents  were  Identified  during  these  efforts  as 
potentially  relevant.  These  were  further  screened  according  to  the 
currency  and  completeness  of  Information  provided  and  the  integrity  of 
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the  experimental  and  analytic  methods  used.  As  a  result  of  this 
screening,  approximately  400  documents  were  found  to  be  useful  for  STRES 
purposes . 

Site  Visits 

A  considerable  body  of  information  also  was  obtained  from 
organizations,  both  government  and  corrmercial ,  whose  personnel  are 
involved  in  the  design,  procurement,  evaluation,  management,  and  use  of 
ATDs.  ATD  manufacturers,  research  and  development  agencies,  and  a 
commercial  airline  were  visited  in  addition  to  Air  Force,  Army,  Navy, 
and  Coast  Guard  military  training  sites.  At  each  organization, 
extensive  data  were  obtained  through  observations,  interviews,  and 
docunent  reviews.  The  training  organizations  visited  and  the  topics  of 
primary  interest  at  each  are  listed  in  Table  2.  Table  3  lists 
non-training  organizations  that  were  visited  and  corresponding  interest 
topics. 

Specific  objectives  of  the  interviews  and  other  data  collection 
efforts  varied,  depending  on  the  type  of  organization  visited  and  the 
purpose  of  the  visit.  Manufacturers  and  research  and  development 
agencies  were  visited  to  assess  current  and  projected  technology  and  to 
review  ongoing  and  planned  efforts  bearing  on  STRES  program  objectives. 
ATD  using  organizations  were  visited  to  obtain  a  variety  of  information 
related  to  types  and  effectiveness  of  training  accomplished,  uses  of 
various  types  of  devices  in  accomplishing  the  training,  ATD  design 
characteristics,  worth  of  ATD  ownership,  and  ATD  life  cycle  costs. 
Detailed  interview  guides  were  used. 

STRES  PHASE  II  REPORTS 

Seven  reports  were  prepared  to  docunent  Phase  II  efforts  and 
findings.  They  are: 

Semple,  C.A.  Simulator  Training  Requirements  and  Effectiveness 
Study  (STRES):’  Executive  Summary.  AFHRL-7R-80-63.  Wright- 
Patterson  AFB,  OH:  Logistics  and  Technical  Training  Division, 

Air  Force  Human  Resources  Laboratory,  January  1981. 

Semple,  C.A.,  Hennessy,  R.T.,  Sanders,  M.S.,  Cross,  B.K.,  Beith, 

B.H.,  &  McCauley,  M.E.  Aircrew  Training  Devices:  Fidelity 
Features .  AFHRL-TR-80-36.  Wright- Patterson  AFB,  OH: 

Logistics  and  Technical  Training  Division,  Air  Force  Human 
Resources  Laboratory,  January  1981. 

Semple,  C.A.,  Cotton,  J.C.,  &  Sullivan,  D.J.  Aircrew  Training 
Devices:  Instructional  Support  Features.  AFHRL-TR-80-58. 

Wright- Patterson  AFB,  OH:  Logistics  and  Technical  Training 
Division,  Air  Force  Human  Resources  Laboratory,  January  1981. 


Table  2.  Training  Sites  Included  In  Program 
Surveys 


Sites  and  Units 

Topics  of  Interest 

Altus  AFB,  OK  (MAC) 

443rd  Military  Airlift  Wing 

C-5  transition  training 

Castle  AFB,  CA  (SAC) 

93rd  Bomb  Wing 

KC-135/B-52  transition  training 

Denver,  CO 

United  Air  Lines 

Flight  Training  Center 

DC-1 0/B-737/B-747  transition 
and  continuation  training 

Eglin  AFB,  FL  (TAC) 

33rd  Tactical  Fighter  Wing 

F-4  continuation  training 

Fort  Rucker,  AL 

US  Army  Aviation  Center 

UH-l/CH-47  undergraduate  and 
transition  training 

Langley  AFB,  VA  (TAC) 

1st  Tactical  Fighter  Wing 

F-15  continuation  training 

Mobile,  AL 

US  Coast  Guard 

Aviation  Training  Center 

HH-3/HH-52  transition  and 
continuation  training 

NAS  Cecil  Field,  FL 

VA-174  and  Light  Attack 

Air  Wing  One 

A-7E  transition  and 
continuation  training 

NAS  Jacksonville,  FL 

VP- 30  and  Patrol  Wing  Eleven 

P-3C  transition  and 
continuation  training 

Plattsburgh  AFB,  NY  (SAC) 

380th  Bomb  Wing 

FB-111  transition 
training 

Reese  AFB,  TX  (ATC) 

64th  Flying  Training  Wing 

T-37/T-38  undergraduate 
pilot  training 

Tinker  AFB,  OK  (TAC) 

552nd  Airborne  Warning  and 

Control  Wing 

E-3A  transition  and 
continuation  training 
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Table  3.  Sites  Visited  For  Management,  Research, 
Development,  Engineering  and  Cost  Surveys 


Sites  and  Agencies 


Topics  of  Interest 


Pentagon 

Headquarters,  USAF 

Randolph  AFB 
Headquarters,  ATC 


Langley  AFB,  VA 
Headquarters,  TAC 


Eglin  AFB,  FL  (TAC) 

Tactical  Air  Warfare  Center 

Luke  AFB,  AZ  (TAC) 

4444th  Operational  Training 
Development  Squadron 

Williams  AFB,  AZ 

Air  Force  Human  Resources 

Laboratory  ( AFHRL/OT ) 

Wright-Patterson  AFB,  OH 
Air  Force  Human  Resources 
Laboratory  (AFHRL/LR ) 

Fort  Rucker,  AL 
US  Army  Research  Institute 
for  the  Behavioral  and 
Social  Sciences 

NASA  Langley  Research  Center 
Langley,  VA 

McDonnell  Douglas  Corp. 

St.  Louis,  MO 

Singer-Link  Corp. 
Blnghampton,  NY 


Management  of  Air  Force  ATD 
resources,  and  life  cycle  costs 

Management  of  the  use  of  ATDs 
In  undergraduate  pilot  training, 
and  life  cycle  costs 

Management  of  the  use  of  ATDs 
In  fighter  aircrew  training, 
development  of  ATD  requirements, 
and  life  cycle  costs 

Procurement,  development  and 
evaluation  of  ATDs 

Development  of  training  and 
ATD  requirements 


ATD  research 


ATD  research 


ATD  research 


ATD  research 


ATD  design  and  research 


ATD  design,  procurement  and 
evaluation 


Navy  Training  Analysis  and 
Evaluation  Group 
Orlando,  FL 


ATD  research  and  life  cycle 
costs 


Table  3.  -  (Continued) 


Sites  and  Agencies 

Topics  of  Interest 

Naval  Training  Equipment 

Center,  Orlando,  FL 

ATD  research  and  life 
costs 

cycle 

Navy  Personnel  Research 
and  Development  Center 

San  Diego,  CA 

ATD  research  and  life 
costs 

cycle 

US  Army  Project  Manager  for 

Training  Devices  (PM-TRADE) 

Orlando,  FL 

ATD  research  and  life 
costs 

cycle 

Hill  AFB,  OK  (AFLC) 

ATD  life  cycle  costs 

Holloman  AFB,  NM  (AFTEC) 

ATD  life  cycle  costs 

Luke  AFB,  AZ  (TAC) 

ATD  life  cycle  costs 

Offutt  AFB,  NE  (SAC) 

ATD  life  cycle  costs 

Scott  AFB,  IL  (MAC) 

ATD  life  cycle  costs 

Travis  AFB,  CA  (MAC) 

ATD  life  cycle  costs 

Williams  AFB,  AZ  (ATC) 

ATD  life  cycle  costs 

Wrlght-Patterson  AFB,  OH  (ASO) 

ATD  engineering  and  life 
cycle  costs 

Caro,  P.W.,  Shelnutt,  J.B.,  &  Spears,  W.D.  Aircrew  Training 
Devices:  Utilization.  AFHRL-TR-80-35.  Wright- Patterson 
AFB,  OH:  Logistics  and  Technical  Training  Division,  Air 
Force  Human  Resources  Laboratory,  January  1981. 

Allbee,  K.E.,  &  Semple,  C.A.  Aircrew  Training  Devices:  Life 

Cycle  Cost  and  Worth  of  Ownership.  AFHRL-TR-80-34.  Wright- 
Patterson  AFB,  OH:  Logistics  and  Technical  Training  Division, 
Air  Force  Human  Resources  Laboratory,  January  1981. 

Prophet,  W.W.,  Shelnutt,  J.B.,  R  Spears,  W.D.  Simulator  Training 
Requirements  and  Effectiveness  Study  (STRESS:  Future  Research 
Plans.  AFHRL-TR- 80-37.  Wright- Patterson  Ah',  OH:  Logistics 
and  Technical  Training  Division,  Air  Force  Human  Resources 
Laboratory,  January  1981. 


Spears,  W.D.,  Sheppard,  H.J.,  Roush,  M.D.,  &  Rlchettl,  C.L. 
Simulator  Training  Requirements  and  Effectiveness  Study 
(SllRES V:  Abstract  Bibliography.  AFHRL-TR-80-3R.  Wrlght- 
Patterson  AFB,  OH:  Logistics  and  Technical  Training 
Division,  Air  Force  Human  Resources  Laboratory,  January  1981. 


The  content  of  the  first  four  of  these  reports,  i.e.,  ATD  fidelity, 
instructional  features,  utilization,  and  cost  and  worth  of  ownership,  is 
interrelated.  As  an  aid  to  the  reader  In  accessing  related  information, 
these  four  reports  were  cross-referenced.  Within  a  single  volune,  other 
chapters  where  related  Information  Is  discussed  are  referenced.  When 
the  cross-referenced  information  is  in  another  volume,  that  volune  is 
Identified  by  abbreviated  title  (Fidelity,  Instructional  Features, 
Utilization,  or  Cost)  as  well  as  by  chapter.  For  example,  Utilization, 
Chapter  IV,  would  Indicate  that  related  information  will  be  found  in 
Chapter  IV  of  the  report  titled  Utilization  of  Aircrew  Training  Devices. 
As  an  additional  aid  to  the  reader,  the  Executive  Summary  volune 
reproduces  the  tables  of  content  of  all  four  technical  volunes  to 
provide  a  consolidated  listing  of  topics  addressed  In  each. 

APPROACH  TO  THE  INSTRUCTIONAL  SUPPORT  FEATURES  SURVEY  EFFORT 

Literature  Review 


Articles  identified  during  the  literature  search  were  screened  for 
relevance  to  ATD  instructional  support  features.  The  screening  was  not 
restricted  to  the  Immediate  domain  of  this  report,  however,  for  the 
perspective  needed  to  answer  questions  related  to  feature  design  and  use 
frequently  required  knowledge  of  broader  Issues  such  as  the  nature  of 
the  training  being  undertaken,  student  entry  level  skills,  and 
Instructional  feature  technology.  Also,  Information  concerned  primarily 
with  phenomena  and  principles  of  learning  and  related  Instructional 
practices  was  needed  for  a  comprehensive  assessment  of  ATD 
instructional  support  features. 


28 


Site  Visit  Activities 


Activities  of  study  team  members  responsible  for  the  instructional 
support  feature  area  during  visits  to  sites  identified  in  Table  2 
Included  Inspection  of  available  simulators  and  related  training  aids* 
and  observation  of  demonstrations  of  pertinent  aspects  of  their 
capabilities  and  use.  The  majority  of  time  was  spent,  however,  in 
intensive  Interviews  with  Instructors  Involved  in  ATD  use,  ATD  console 
operators,  and  pilots  undergoing  training.  Additionally,  maintenance 
personnel  were  interviewed  on  the  reliability  of  ATD  consoles  and 
instructional  features.  A  detailed  interview  guide  (Appendix  A)  was 
used,  and  notes  were  recorded  during  and  following  the  interviews. 

Detailed  information  relating  specifically  to  instructional  support 
features  Included:  1)  training  objectives;  2)  training  methods  and 
techniques  used;  3)  types  of  ATDs  used  In  training  each  objective;  4) 
the  roles  of  instructors  and  ATD  console  operators;  5)  Instructional 
support  features  available  in  the  ATDs;  6)  instructor  and  operator 
training  in  the  use  of  available  features;  7)  desirable  and  undesirable 
feature  design  characteristics;  and  8)  types  and  extents  of  their  use 
for  the  training  being  provided.  As  shown  in  the  interview  guide, 
information  pertaining  to  the  overall  training  programs  was  also 
acquired  so  that  utilization  practices  involving  available  instructional 
support  features  could  be  put  Into  perspective. 

Interviews  with  key  personnel  were  the  principal  Information 
gathering  technique  used  during  visits  to  the  training  sites  indicated 
in  Table  2.  Interview  guides  used  at  the  training  sites  served  as 
checklists  for  topics  to  be  addressed  at  each  agency.  The  thrust  of 
visits  to  sites  shown  in  Table  3,  however,  was  to  obtain  Information 
about  research  In  progress  or  planned,  advanced  simulation  technology, 
present  and  anticipated  regulatory  requirements,  and  policies  that  could 
contribute  to  improved  future  AID  training  programs.  Information  gained 
from  these  sources  also  broadened  the  perspective  from  which  practices 
at  training  sites  could  be  viewed. 


CHAPTER  III 


INSTRUCT 


FUNCTION  DEFINITION 

The  primary  reason  for  using  an  ATD  is  to  instruct.  Other  uses 
include  proficiency  checks  and  research.  The  formal  definition  of 
instruct  is  to  impart  knowledge  and/or  skill  in  a  systematic  fashion. 

Instruct  is  a  broad  functional  concept  that  has  different  meanings 
to  different  people  and  can  incorporate  wide  sets  of  training  activities 
and  techniques.  For  the  purpose  of  this  chapter,  instruct  is  defined  as 
consisting  of  four  highly  interrelated  sub- functions:  1)  providing 
correct  procedures;  2)  providing  technique  information;  3)  critiquing; 
and  4)  providing  feedback  Information. 

RELATED  FUNCTIONS 

The  reader  is  directed  to  the  following  chapters  that  address 
functions  related  to  the  instruct  function: 

Chapter  IV,  Monitor  and  Evaluate  Performance 

Chapter  V,  Control  and  Individualize  Training 


INSTRUCTIONAL  SUPPORT  FEATURES  ADDRESSED 

The  instructional  features  addressed  in  this  chapter  are: 
Freeze 

Automated  demonstration 
Record  and  replay 
Automated  cuing  and  coaching 

Manual  and  programmable  sets  of  Initializing  conditions 
Manual  and  programnabl e  malfunction  control 
ATO-mounted  audio  visual  devices 
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FREEZE 


Summary 

Freeze  Is  a  common  ATD  instructional  feature.  It  allows  the 
instructor/operator  to  stop  all  or  part  of  ATD  system  operations  by 
fixing  the  value  of  one  or  more  computational  parameters.  Seven 
variations  on  the  basic  freeze  capability  were  identified  during  program 
site  visits.  Instructional  values  of  the  feature  depend  on  the 
variation  being  used.  Assumed  instructional  values  include:  reduce  or 
otherwise  control  trainee  task  loading;  simplify  flight  control  and 
related  procedural  tasks;  enable  more  productive  instructor* student 
Interaction;  and  enable  OFTs,  PTTs  and  FMTs  to  be  used  for  procedures 
training  when  other  means  are  not  available.  Other  potential  values  are 
discussed  in  this  section.  Relationships  are  presented  between 
alternative  freeze  capability  designs  and  intended  uses.  It  is 
concluded  that  four  variations  of  freeze  (total  system  freeze,  flight 
system  freeze,  position  freeze,  and  parameter  freeze)  are  sufficient 
for  practically  all  aircrew  training  applications. 

Feature  Description 

The  freeze  feature  allows  simulator  parameters  to  be  fixed  at  values 
existing  when  freeze  Is  operated.  Some  or  all  device  parameters 
associated  with  the  student's  task  can  be  fixed.  The  freeze  feature  can 
be  operated  manually  by  the  instructor,  or  it  can  be  operated 

automatically  when  specified  parameter  values  are  exceeded,  as  in  the 
case  of  "crash".  The  feature  and  its  variations  have  the  greatest 

potential  application  in  OFTs,  PTTs  and  FMTs.  Observed  variations  of 
the  freeze  feature  are  defined  below  to  provide  standard  definitions  for 
use  in  subsequent  discussions. 

Total  System  Fry ze.  All  system  parameters  are  frozen,  including 
flight  control,  propulsion,  navigation  and  weapons.  The  entire 
simulation  ceases  to  function  from  a  training  standpoint.  For 
systems  equipped  with  platform  motion,  the  motion  system  is  driven 
to  a  neutral  position,  at  a  safe  rate,  allowing  for  exit  from  or 
entry  into  the  ATD. 

Flight  System  Freeze.  Flight  control  and  propulsion  system 
parameters  are  frozen,  together  with  latitude,  longitude,  altitude 
and  heading.  The  net  effect  is  that  the  ATD  ceases  to  "fly". 
However,  all  other  simulated  systems  remain  operational  and  can  be 
used  for  instructional  purposes  without  the  burden  of  having  to  fly 

the  ATD.  This  freeze  variation  functionally  converts  an  OFT  or  FMT 

to  a  CPT. 

Automatic  Freeze.  This  variation  of  freeze  commonly  is  called 
"crash11.  Crash  occurs  automatically  when  specified  parameter 
values  are  exceeded.  Automatic  freeze,  as  commonly  Implemented, 
causes  all  ATD  parameters  to  reinitialize  to  specified  values,  thus 
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interrupting  the  mission.  For  systems  equipped  with  platform 
motion,  the  motion  system  is  driven  to  a  neutral  position  at  a  safe 
rate. 

Crash  Override.  Crash  override  disables  automatic  freeze. 

Kill .  This  freeze  variation  is  similar  to  but  not  identical  with 
crash.  Kill  can  be  operated  either  automatically  or  manually  to 
indicate  a  lethal  weapon  hit.  Cockpit  Indications  and  motion 
system  operations  are  identical  with  crash.  In  most  applications, 
however,  the  ATOs  software  Is  progranned  to  freeze  all  operational 
parameters,  rather  than  automatically  reinitialize  them  to 
predetermined  values. 

Kill  Override.  Kill  override  unfreezes  system  parameters  frozen  by 
the  kill  function.  This  enables  the  mission  to  continue  without 
the  need  to  reinitialize  the  ATD. 

Parameter  Freeze.  This  variation  of  freeze  enables  the  Instructor 
to  fix  the  value  of  individual  parameters,  usually  from  the 
Instructor/operator  console.  The  values  of  one  or  more  parameters 
can  be  frozen  at  any  given  time.  Typical  uses  include  freezing 
just  altitude,  heading  or  the  roll  axis  of  flight  control. 

Attitude  Freeze.  This  is  a  variation  of  parameter  freeze  where 

val ues  for  pitch ,  bank  and  heading  are  frozen  simultaneously. 

Latitude  and  longitude  may  or  may  not  be  frozen,  depending  on  the 
Implementation.  Motion  platform  position  also  is  frozen  at  the 
angular  values  existing  when  attitude  freeze  Is  selected,  at  least 
in  present  implementations. 

Position  Freeze.  This  variation  freezes  values  of  latitude  and 

longitude  only.  Thus,  the  ATD  continues  to  "fly",  but  it  "goes 

nowhere"  geographi cal 1 y . 

Related  Support  Features 

Relationships  of  freeze  to  other  instructional  support  features  are 
summarized  below.  All  relationships  were  observed  during  site  visits. 

Record  and  Replay.  A  number  of  ATDs  have  the  capability  to  record 
parameter  values  and  use  these  data  to  recreate  system  behavior  for 
students  and  Instructors  to  use  in  a  review  mode.  The  use  of  total 
system  freeze  Is  common  In  devices  during  replay.  It  has  proved 
useful  to  stop  the  replay  to  reduce  the  task  load  on  the  student  to 
allow  for  Improved  performance  problem  diagnosis  and  Instruction, 
and  simply  to  stop  the  replay  to  allow  reinitializing  the  ATD.  (See 
the  record  and  replay  section  of  this  chapter) 
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Automated  Demonstrations.  Automated  demonstrations  permit  the 
standardized  presentation  of  a  mission  segment  or  an  entire 
simulated  flight.  As  in  replay,  the  ability  to  system  freeze  has 
proved  useful  to  stop  the  demonstration  to  allow  for  student 
questioning  or  instruction,  and  simply  to  stop  the  demonstration  to 
allow  reinitializing  the  ATD.  (See  the  section  of  this  chapter  on 
manual  and  programnabl e  sets  of  initializing  conditions) 

Initialize  ATD.  Total  system  freeze  was  used  in  all  ATDs  surveyed 
to  stop  an  ongoing  exercise  as  the  first  step  in  reinitializing  the 
ATO.  (See  the  section  of  this  chapter  dealing  with  manual  and 
programmable  sets  of  Initial  conditions) 

Instructional  Values 


Three  instructional  values  commonly  are  assumed  for  the  freeze 
feature.  The  most  common  is  to  provide  for  more  productive  instructor- 
student  interaction.  Total  system  freeze  and  flight  system  freeze 
support  this  use,  because  many  task  requirements  are  removed  from  the 
student's  task  load.  This  often  provides  for  improved  Instructional 
opportunities.  Also,  total  system  and  flight  system  freeze  variations 
provide  a  stable  (non-moving)  crewstation  environment  which,  for 
example,  allows  the  Instructor  to  stand  immediately  outside  an  ATD 
cockpit,  with  the  canopy  open,  but  without  an  unacceptable  safety 
hazard.  Uses  of  attitude  freeze  or  position  freeze,  in  combination  with 
parameter  freeze,  were  observed  during  the  program  as  ways  of  achieving 
flight  system  freeze  for  the  above  purposes. 

A  second  instructional  value  Is  the  reduction  of  student  task 
loading  In  cases  where  the  Instructor  may  not  be  present  or  elects  not 
to  Intervene  directly.  Flight  system  freeze  and  position  freeze  support 
this  ATD  use.  For  example,  either  the  Instructor  or  the  student  may 
request  a  temporary  halt  In  the  ATD  training  session  so  the  student  can 
review  checklists,  charts  or  other  necessary  documents;  reorient  himself 
geographically;  or  discuss  crew  roles  and  coordination  with  other 
crewmembers.  Taking  a  rest  break  also  is  a  proper  use  of  freeze. 

A  third  instructional  value  is  that  simplifying  flight  control  tasks 
during  Initial  training  may  speed  the  learning  of  new  flight  control 
skills.  Parameter  freeze  is  suited  to  the  control  of  individual  flight 
parameters  or  axes.  For  example,  altitude  can  be  frozen  while  training 
Is  focused  on  bank  and  heading  control.  Similarly,  heading  can  be 
frozen  while  training  Is  focused  on  pitch  and  power  control.  No 
experimental  evidence  is  known  that  either  supports  or  refutes  the 
effectiveness  of  using  parameter  freeze  in  this  manner  for  fixed  wing 
pilot  training.  At  the  time  this  report  was  written.  Instructor 
experience  with  this  use  of  parameter  freeze  was  insufficient  to  allow 
the  forming  of  stable  opinions. 
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The  following  Instructional  principles,  supported  by  research,  are 
involved  in  the  assumed  values  and  proper  uses  of  freeze. 

Control  cues  to  ensure  that  the  student  is  forming  proper 
associations;  i.e.  responding  appropriately  to  the  correct  cues. 
Freeze  can  be  used  to  teach  and  reinforce  correct  associations. 

Stimuli  used  in  training  should  be  nearly  identical  to  job  stimuli 
unless  this  fidelity  increases  problem  difficulty  in  initial  phases 
of  training  to  unacceptable  (unproductive)  levels.  Parameter, 
position  or  attitude  freeze  can  simplify  problem  difficulty. 

Emphasize  differences  between  correct  and  incorrect  cue  forms. 


Three  additional  instructional  values  of  the  freeze  feature  were 
observed  during  the  program.  Their  applications  are  more  situation 
specific  than  the  ones  discussed  above. 

The  first  application  Is  termed  controlled  demonstration.  In  the 
general  case,  flight  parameters  are  selectively  frozen  to  limit  the 
progress  of  the  simulation  In  time  and  space.  For  example,  position 
freeze  may  be  selected  so  that  the  ATD's  geographic  reference  is  fixed 
during  a  final  approach  to  a  visual  landing.  The  Instructor  could  then 
use  pitch  and  power  control  to  demonstrate  runway  visual  scene 
perspectives  for  a  properly  executed  approach,  or  an  approach  that  is 
too  high  or  too  low.  This  use  of  freeze  should  benefit  the  student's 
learning  to  distinguish  visual  cues  associated  with  an  acceptable 
approach  from  those  of  an  unacceptable  approach.  The  advantages  are 
that  the  cues  are  presented  systematically  by  the  instructor,  and 
students  can  systematically  learn  differences  between  correct  and 
Incorrect  cue  patterns.  It  is  not  necessary  to  let  the  student  make 
mistakes  In  order  to  learn  the  cues  associated  with  the  mistakes. 

The  second  application  Involves  training  resource  management. 
Occasionally  it  is  necessary  to  share  resources  among  ATOs.  Position 
freeze  can  aid  this  sharing.  For  example,  some  ATDs  share  model  board 
visual  capabilities.  Position  freeze  can  be  used  to  temporarily  "halt" 
the  flight  of  one  ATO  while  another  uses  the  available  visual  system. 
Similarly,  some  ATO  complexes  require  one  controller  to  control  several 
simulated  flights.  Particularly  during  ground  controlled  approaches.  It 
Is  convenient  and  Instructional ly  sound  to  position  freeze  some  ATOs 
while  another  is  given  precision  control  Information. 

A  third  application  of  freeze  Is  to  create  a  procedures  training 
environment  in  an  OFT,  PTT  or  FMT.  A  common  criticism  Is  that  OFTs  and 
FMTs  frequently  are  "misused"  as  procedures  trainers.  However,  if  a 
separate  procedures  trainer  Is  not  available,  or  if  it  is  Instruction- 
ally  necessary  to  temporarily  Interrupt  ongoing  training  for  remedial 
procedures  training  that  is  needed  for  productive  use  of  remaining 


training  time,  then  freezing  position  variables  and  flight  control 
parameters  Is  highly  desirable. 

Observed  Applications 


The  freeze  feature  was  present  in  various  forms  in  all  of  the  OFTs, 
PTTs  and  FMTs  observed  during  the  program.  Table  4  presents  a  summary 
of  the  variations  of  freeze  that  were  observed.  Additionally, 
variations  of  freeze  are  being  Incorporated  into  the  following  ATD 
designs:  A-10;  B-52;  C-130;  F-5;  and  F-16. 

Many  instructional  features.  Including  freeze,  cannot  be  directly 
related  to  specific  task  training  requirements.  This  is  because 
features  like  freeze  primarily  are  means  to  other  ends.  Such  means 
frequently  are  not  unique  to  the  tasks  being  trained.  Rather,  they 
involve  the  application  of  Instructional  methods  and  procedures.  The 
instructional  feature.  In  this  case,  provides  an  important  capability. 
The  capability  is  to  enable  the  use  of  appropriate  instructional  methods 
and  procedures.  For  example,  the  need  to  interrupt  a  training  session 
for  a  rest  break  Is  not  unique  to  the  tasks  being  trained.  Similarly, 
the  need  to  reduce  student  task  loading  distractions  while  diagnosing  a 
performance  deficiency  and  prescribing  correct  procedures  or  techniques 
is  not  unique  to  specific  training  tasks.  The  issues  are  addressed 
further  in  the  section  titled  design  considerations. 

Utility  Information 

The  frequency  of  use  of  freeze  capabilities  varied  significantly 
among  the  sites  visited.  This  was  due  to  differences  in  student  skill 
levels  and  to  differences  in  instructional  practices.  Freeze  was  used 
more  frequently  when  the  tasks  being  learned  by  students  in  ATDs  were 
relatively  or  totally  new  to  them.  Also,  the  use  of  freeze  hinged  on 
the  use  of  other  Instructional  support  capabilities,  such  as  automated 
demonstrations,  record  and  replay,  and  ATD  reinitialization. 

Total  system  freeze  Is  used  most  frequently.  It  is  the  common  means 
for  stopping  simulations  for  entry  and  exit,  and  for  starting  the 
simulation  when  all  crewmembers  are  at  their  positions.  Total  system 
freeze  also  is  used  to  temporarily  stop  automated  demonstrations  or 
replays  for  verbal  Instruction,  and  to  terminate  them  to  reinitialize 
the  ATD.  Total  system  freeze  also  is  necessary  for  ATD  maintenance. 

Flight  system  freeze  was  used  most  frequently  during  undergraduate 
and  transition  training.  The  instructional  use  was  to  relieve  students 
from  flight  control  task  loads  so  that  other  Instructional  activities 
could  take  place  in  a  more  favorable  learning  environment.  Maintenance 
and  evaluation  uses  were  to  stop  ATD  flight  progress  to  calibrate  and  to 
make  measurements.  Cases  were  observed  where  position  freeze,  attitude 
freeze  and  parameter  freeze  combinations  had  to  be  used  to  cause  the 
same  effects  produced  by  flight  system  freeze. 
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Table  4.  Surveyed  ATDs  Equipped  With  Freeze  Feature 


Type  of  Training 


Undergraduate 
Pilot  Training 


Transition  & 
Continuation 
Training 


Aircraft  Simulated  Freeze  Capabilities 


T-37/T-38 

Total  system,  position, 
parameter,  and  auto¬ 
matic 

UH-1 

Total  system, 
parameter,  automatic 

F-15 

Total  system,  position 
parameter,  automatic, 
crash  override,  kill, 
kill  override 

F-14 

Total  system, 
automatic,  crash 
override 

A-7E  (NCLT) 

Total  system 

FB-111 

Total  system,  position 
parameter,  automatic, 
crash  override 

B-52 

Total  system,  position 
automatic,  crash  over¬ 
ride 

HH-53 

H-3 

Total  system, 
parameter,  automatic, 
crash  override 

P-3C 

Total  system,  position 
automatic,  crash  over¬ 
ride,  parameter 

E-3A 

Total  system,  position 
automatic,  crash 
override,  parameter 
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Table  4.  (Continued) 


Type  of  Training 

Aircraft  Simulated 

Freeze  Capabilities 

C-5 

Total  system,  attitude, 
automatic,  crash 
override 

B-747 

Total  system,  attitude. 

DC-10 

position,  parameter, 
automatic,  flight 
system 
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Automatic  freeze  (crash)  is  common.  Instructors  and  console 
operators  indicated  a  strong  preference  for  crash  override  as  a 
companion  freeze  variation.  This  was  due  to  the  fact  that  all  crash 
implementations  that  were  observed  caused  the  ATD  to  reinitialize  to  a 
predetermined  set  of  conditions  or  required  instructor/operator 
intervention  to  reinitialize  the  ATD  following  crash.  Crash  override 
disabled  automatic  freeze  and  allowed  the  training  exercise  to  continue 
without  the  need  to  reinitialize.  Crash  override  was  viewed  as  a  very 
desirable  feature  for  this  reason. 

The  kill  variation  of  automatic  freeze  was  observed  in  only  one  ATD. 
In  that  application,  kill  froze  ATD  parameters  at  the  values  existing  at 
weapon  Impact.  Thus,  reinitializing  was  not  Involved.  This  was  viewed 
favorably  by  instructors.  The  complementary  feature,  kill  override,  was 
used  to  unfreeze  following  kill  or  to  override  kill  and  avoid  freezing. 
Kill  override  was  viewed  as  a  very  desirable  feature. 

Position  freeze  was  found  to  be  a  useful  tool  to  stop  the  progress 
of  simulated  flight  so  that  visual  system  model  boards  could  be  shared 
among  ATDs  and  so  that  a  single  console  operator  could  act  effectively 
as  a  GCA  controller  for  groups  of  ATDs.  Neither  of  these  goals  could 
have  been  achieved  conveniently  and  unobtrusively  without  position 
freeze.  The  acceptance  of  this  freeze  variation  also  is  indicated  by 
user  investigations  of  It  as  a  tool  for  demonstrating  ground  scene 
perspectives  for  maneuvers  involving  closure  with  the  ground. 

Observed  instructional  uses  of  attitude  freeze  were  to  create  CPT 
environments  in  OFTs  and  FMTs.  Depending  on  the  attitude  freeze 
implementation,  parameter  freeze  (e.g.  altitude)  also  had  to  be  used  In 
some  ATDs  to  achieve  the  desired  flight  system  freeze  effect.  When 
training  objectives  and  needs  required  a  "CPT  mode",  instructor 
acceptance  of  being  able  to  achieve  the  mode  was  high.  Attitude  freeze, 
with  or  without  parameter  freeze,  also  was  used  by  maintenance  personnel 
for  troubleshooting,  calibration  and  system  checkout  in  both  military 
and  commercial  operations. 

Uses  of  parameter  freeze  in  fixed  wing  ATDs,  other  than  uses 
described  above,  were  solely  for  maintenance  or  maintenance  avoidance 
purposes.  Fixed  wing  instructors  who  were  interviewed  never  used 
parameter  freeze  to  simplify  flight  control  tasks  by,  for  example, 
freezing  heading  so  that  training  could  focus  on  vertical  axis  control. 
The  reason  sited  was  the  departure  from  what  can  be  done  In  the  air. 
This  probably  is  not  a  valid  position,  at  least  for  early  flight 
training. 

Related  Research 

No  experimental  research  was  found  that  systematically  dealt  with 
the  design  or  use  of  ATD  freeze  capabilities.  However,  Hughes  (1979) 
independently  arrived  at  many  of  the  conclusions  presented  In  this 


section  based  on  training  research  observations  and  applications  of 
instructional  principles. 

Design  Considerations 

Design  considerations  for  any  instructional  support  feature* 
including  freeze,  must  center  on  intended  uses.  One  apparent  trend  in 
ATD  design  is  toward  expanded  flexibility.  For  freeze,  the  ultimate 
flexibility  is  achieved  by  freezing  combinations  of  Individual 
parameters.  While  some  training  applications  might  benefit  from  the 
ability  to  Individually  freeze  ATD  parameters,  using  parameter  freeze  to 
accomplish  all  desired  freeze  effects  would  be  cumbersome  for  the 
i nstructor/operator . 

A  far  better  approach  is  to  identify  the  uses  to  which  freeze  will 
be  put  to  support  training  needs,  and  then  to  Incorporate  Into  ATDs  the 
freeze  variations  that  meet  those  needs.  This  approach  simplifies 
instructor/operator  tasks  while  ensuring  meaningful  control  over  the 
training  environment.  The  following  design  guidelines  present  uses  of 
freeze  together  with  the  freeze  variations  that  best  match  each  use. 
Definitions  of  freeze  variations  are  those  presented  at  the  beginning  of 
this  chapter. 


Uses 

Start  and  stop  ATD  to  enter 
and  exit,  temporarily 
interrupt  automated 
demonstrations  or  replays, 
or  prepare  to  reinitialize. 

Eliminate  flight  control  and 
navigation  task  loading  to 
better  enable  systems  and 
procedures  training. 

Enable  geographic  reorientation 
of  the  student,  demonstrate 
visual  scene  perspectives,  or 
tlmeshare  training  resources 
(e.g.  model  boards  or  traffic 
controllers). 

Exercise  control  over  the  number 
of  axes  of  flight  to  be  controlled 
simultaneously. 


Freeze  Variations 


Total  system  freeze 


Flight  system  freeze 


Position  freeze 


Parameter  freeze 
(specified  parameters) 


Maintenance  troubleshooting, 
calibration  and  evaluation. 


All  of  the  above 


Maintenance  uses  of  freeze  were  not  investigated  indepth  during  the 
program.  The  reader  is  advised  to  seek  independent  guidance  on  the  uses 
of  freeze  for  maintenance. 

AUTOMATED  DEMONSTRATIONS 

Summary 

Automated  demonstration  (auto-demo)  in  an  ATD  permits  the  device  to 
reproduce  a  previously  established  segment  or  phase  of  flight  without 
direct  Intervention  by  the  student  or  the  instructor.  It  is  commonly 
assumed  that  the  instructional  value  of  auto-demo  lies  in  the 
presentation  of  a  performance  model  that  the  student  can  observe  and 
analyze  to  guide  his  own  task  performance.  To  date,  the  feature  has 
been  limited  to  more  sophisticated  ATDs.  Design  of  this  feature 
requires  consideration  of  the  types  of  training  the  ATD  Is  Intended  to 
support,  and  the  difficulties  of  auto-demo  construction  prior  to  the 
decision  to  Incorporate  the  feature  in  an  ATD  design.  For  example.  It 
is  likely  that  auto-demo  has  its  greatest  value  when  the  student  is 
unfamiliar  with  the  basic  task  or  aircraft- unique  specifics  of  the  task 
he  is  learning. 

Description 


Auto-demo  permits  the  standardized  presentation  of  a  mission  segment 
or  entire  simulated  flight.  All  cues  of  consequence,  including  the 
visual  scene  (if  present),  motion  system,  primary  flight  controls  and 
displays,  crew  communications,  and  sensor  displays  are  reproduced  for 
the  selected  mission  segment  or  maneuver.  Through  computer  control,  the 
ATD  operates  as  though  skilled  aircrew  members  were  at  the  controls. 

Auto-demo  requires  substantial  two-way  communication  between  ATD 
controlling  software  and  hardware,  and  crew  station  controls  and 
displays.  Given  this  requirement,  auto-demo  Is  found  only  in  more 
sophisticated,  computer  controlled  ATDs,  such  as  OFTs,  PTTs,  and  FMTs. 
ATD  hardware  and  software  must  be  configured  to  Induce  changes  in  the 
configuration/operation  of  the  simulated  subsystems  without  any 
intervention  on  the  part  of  the  student  or  the  Instructor. 

Technically,  auto-demo  can  be  constructed  for  any  segment  of  flight. 
Thu$,  for  example,  an  ATD  could,  on  command,  takeoff,  fly  straight  and 
level,  perform  aerobatic  or  air  combat  maneuvers,  deliver  weapons,  fly  a 
standard  approach,  and  land  without  the  student  or  Instructor  operating 
any  primary  controls. 

Related  Support  Features 

The  following  list  Identifies  relationships  to  other  instructional 
features.  Some  of  the  relationships  now  exist  In  ATDs;  others  are 
offered  as  candidates  for  future  ATDs. 
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Record  and  Replay.  The  ability  to  record  the  performance  of  the 
pilot  and  the  aircraft,  and  to  present  that  performance  to  the 
student  and  Instructor  In  a  review  mode  exists  In  a  number  of  ATDs. 
Record  and  replay  has  served  as  one  method  for  creating  auto-demo 
presentations.  The  capability  to  fly-out  of  the  replay  (l.e., 
assume  manual  control)  appears  necessary  for  efficient  creation  of 
auto-demo  presentations  (or  to  hand  control  back  to  the  student). 

Freeze.  Freeze  allows  the  Instructor/operator  to  suspend  some  or 
all  device  parameters  associated  with  the  student's  task.  This  can 
range  from  suspending  all  parameters  (total  system  freeze)  through 
suspending  of  only  one  parameter  (e.g.,  just  pitch  control  or  axis 
freeze).  Where  the  use  of  the  auto-demo  feature  has  been  observed, 
system  freeze  also  has  been  used  to  Interrupt  the  demonstration  for 
verbal  Interactions  with  the  student,  and  to  halt  the  demonstration 
so  the  ATD  can  be  reinitialized.  (See  the  freeze  section  of  this 
chapter) . 

Automated  Briefing.  Automated  (canned)  briefings  offer  highly 
standardized  verbal  commentary  to  accompany  standardized 
demonstrations.  One  Initial  entry  rotary  wing  training  device 
surveyed  incorporated  this  capability.  It  was  judged  to  be 
valuable,  was  used  to  provide  standardized  commentary  during 
automated  demonstrations,  and  was  the  only  available  source  of 
instructional  commentary  during  periods  when  students  used  the 
device  without  the  direct  supervision  of  an  Instructor.  One  airline 
that  was  surveyed  similarly  used  canned  briefing  to  guide  practice 
when  an  Instructor  was  not  present. 

Synthesized  Controller.  Synthesized  speech  (l.e.,  computer 
generated  speech)  represents  a  new  instructional  support  technology 
that,  to  date,  has  not  been  Incorporated  Into  many  ATDs.  If  such  a 
capability  were  Incorporated  (e.g.,  an  automated  GCA  controller)  It 
Is  only  logical  that  the  synthesized  speech  messages  should  be 
Incorporated  as  a  facet  of  a  GCA  or  other  similar  demonstration. 
(See  Chapter  VI,  Controller  Function) 

Instructional  Values 


Two  Instructional  values  commonly  are  assumed  for  auto-demo.  The 
most  common  Is  that  the  feature  provides  a  performance  model  that  the 
student  can  observe,  analyze,  pattern  his  own  behavior  pattern  after, 
and  use  as  a  reference  for  self-evaluation  in  subsequent  training  trials 
or  during  operational  flying.  Associated  with  this  Is  the  fact  that  the 
student's  workload  Is  considerably  reduced  during  auto-demo,  which 
provides  a  better  opportunity  to  observe  relationships  among  cues  and 
their  relationship  to  system  behavior.  Interact  with  the  instructor,  or 
attend  to  an  automated  briefing.  Similarly,  Instructor  workoad  is 
considerably  reduced,  allowing  a  better  opportunity  for  Instructional 
Interaction  with  the  student. 
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A  second  commonly  assumed  value  acknowledges  that  automated 
demonstrations  may  be  the  only  way  to  show  the  student  what  Is  expected 
of  him.  It  has  been  common  practice  in  the  design  of  virtually  all  ATDs 
to  exactly  reproduce  the  aircraft  crewstation  physical  configuration. 
When  the  crewstation  is  either  single  seat  (e.g.,  F-15)  or  incorporates 
controls  and  displays  necessary  to  execute  manual  demonstrations  at  only 
one  crew  position  (e.g.,  F-4),  the  .opportunity  for  an  Instructor  to 
enter  the  crewstation  and  do  a  hands-on  demonstration  is  excluded. 
Therefore,  the  only  remaining  avenue  for  a  demonstration  is  by  means  of 
auto-demo. 

The  following  learning  principles  are  Involved  in  the  instructional 
benefits  described  above  and  in  the  proper  use  of  auto-demo: 

Prevent  decay  of  recall  by  increasing  the  meaningful  ness  of  the 
material  learned,  l.e.,  by  providing  organization  of  related  facts 
and  principles. 

Control  cues  to  make  sure  that  the  student  is  forming  the  proper 
associations,  l.e.,  responding  appropriately  to  the  correct  cues. 

Use  programed  demonstrations  of  procedures  up  to  but  not  beyond  the 
ability  of  the  student  to  understand  the  procedures. 

Emphasize  cues  that  cause  mediating  responses,  e.g., 

self-instruction  and  natural  associations. 

A  number  of  non-tradltional  uses  of  auto-demo  were  identified  during 
the  course  of  the  STRES  program.  No  research  evidence  exists  that 
either  supports  or  refutes  their  value  for  particular  training 
applications.  The  possible  uses  are  presented  here  to  stimulate  thought 
and  promote  research  Into  their  potentials.  They  represent  a  departure, 
in  some  instances,  from  the  flight  training  model.  The  flight  training 
model  is  an  approach  to  training  that  patterns  the  way  aircrew  training 
is  done  In  an  ATD  directly  after  the  way  comparable  training  Is  done  In 
the  aircraft.  The  approach  often  Imposes  limitations  on  ATD  use  that 
stem  from  the  ways  aircraft  must  be  used  for  training. 

Non-Ideal  Performance  Models.  Present  mechanizations  of  the 
auto-demo  provide  a  "perfect"  performance  model  for  the  student  to 
pattern  after.  A  related  use  would  be  to  demonstrate  certain 
aspects  of  undesirable  performance,  such  as  typical  errors  committed 
by  learners  at  various  stages  of  skill  development  and,  even  more 
Important,  the  consequences  of  those  errors.  The  "Ideal" 
performance  model  could  then  be  brought  into  play  to  demonstrate 
accepted  recovery  procedures. 

Tactics  Development.  Related  to  the  performance  model  application 
is  the  use  of  the  auto-demo  feature  as  a  method  to  record  (e.g.  on 
magnetic  tape),  evaluate  and  critique  newly  developed  or  proposed 
tactics.  In  this  application,  the  auto-demo  feature  would  be  used 
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to  present  the  tactical  evaluation  to  individuals  and  organizations 
(e.g.,  RAD  groups,  operational  squadrons,  etc.)  who  are  not  directly 
affiliated  with  the  organization  that  developed  the  tactics.  They, 
in  turn,  could  use  the  demonstration  to  train  their  personnel,  and 
could  use  their  own  record-replay  feature  to  develop  alternative 
sets  of  tactics  that  could  be  returned  (e.g.  on  magnetic  tape)  to 
the  tactics  proponent  for  review  and  comment.  This  kind  of  inter¬ 
change  of  concrete  tactical  presentations  could  accelerate  the 
exploitation  and  adoption  of  improved  tactics  and  provide  the  basis 
for  new  instructional  applications  of  existing  ATDs. 

The  same  tactical  development  dialogue  application  has  potential  for 
comnunl eating  specialized  or  limited  tactics  Information  in  an 
extremely  useful  form.  Consider  the  example  of  a  requirement  for  a 
strike  against  a  specific  target  (base,  troop  concentration,  etc.) 
with  known  defenses  and  very  restricted  (politically  or  physically) 
approach  and  escape  corridors.  The  auto-demo  feature  could  be  used 
to  simultaneously  brief  and  train  at  their  home  bases  all  units  who 
are  to  participate  in  the  strike. 

Introduction  to  Aircraft  Characteristics.  A  different  use  of 
auto-demo  is  to  present  airframe  characteristics  under  normal  and 
abnormal  conditions.  This  allows  the  safe  and  controlled 
presentation  of,  for  example,  normal  versus  asymmetrical  flap 
extension,  the  effects  of  different  types  of  structural  damage,  or 
asymmetrical /hung  external  stores.  It  also  can  be  used  to  Introduce 
the  student  to  stall  and/or  departure  characteristics  of  the 
aircraft.  These  demonstrations  could  Include  all  relevant  visual, 
motion  and  auditory  cues.  Obviously,  a  high  level  of  ATD  flight 
characteristics  fidelity  would  be  required.  (See  the  Fidelity 
Volume,  Chapter  IV) 

Slow  Motion  Demonstrations.  All  auto-demo  capabilities  known  to 
exist  now  are  used  only  to  present  demonstrations  in  real  time. 
Questions  of  continuing  interest  are:  1)  Would  slower  than  real 
time  demonstrations  enable  the  learner  to  better  analyze  cues  and 
responses  associated  with  maneuvers  and  form  better  internal  models 
of  expected  performance?  and  2)  Who  might  benefit  most  from  slow 
motion  demonstrations:  undergraduate,  transitioning  or  continuation 
pilots? 

Answers  to  these  questions  presently  are  not  available  either  from 
operational  experience  or  controlled  research.  One  Initial  entry 
rotary  wing  training  device  was  observed  that  has  a  one-half  time 
replay  capability,  and  hence  a  one-half  time  demonstration 
capability.  This  non-real  time  capability  was  not  used  because  It 
was  "not  realistic".  The  C-130  ATD  which  presently  is  being 
procured  will  Incorporate  real  time,  one-half  time  and  one-quarter 
time  auto-demo  capabilities.  Systematic  use  of  these  existing  or 
soon  to  exist  capabilities  should  be  undertaken  to  provide  Insights 
Into  the  training  value  of  slow  motion  demonstrations.  Similarly, 


controlled  laboratory  experimentation  should  be  undertaken.  Both 
types  of  research  are  needed  to  determine  the  training  value  of  this 
"unrealistic"  approach  and  to  determine  for  which  pilots  group  the 
values  may  exist. 

Capabilities  Demonstration  for  Interested  Visitors.  ATDs  are  not 
used  just  for  training.  Occasionally,  visiting  dignitaries,  senior 
officers,  other  aircrew  personnel  or  engineering  specialists  have 
needs  to  observe  and  discuss  facets  of  training  programs,  devices, 
aircraft  systems,  operational  characteristics  and  device  or  aircraft 
features.  Properly  designed,  auto-demos  could  be  used  to 
demonstrate  aircraft  operating  characteristics,  weapon  system 
capabilities,  device  capabilities  and  learning  strategies  used 
during  training.  Auto-demos  would  ensure  that  nothing  of  relevance 
would  be  overlooked  during  the  demonstration.  In  single  seat  ATDs, 
automation  might  be  the  only  way  of  providing  a  demonstration.  This 
would  be  the  case,  for  example,  If  the  visitor's  skill  level  or 
knowledge  was  Inadequate  to  allow  him  to  effectively  "fly"  the  ATD 
alone.  Pre-recorded  voice  messages  could  be  used  to  ensure 
appropriate  verbal  commentary.  Special  visitor  auto-demos  might  not 
be  required.  Those  used  during  normal  training  operations  might  be 
quite  adequate,  depending  upon  their  content. 

Observed  Applications 

The  auto-demo  feature  was  present  in  several  of  the  ATDs  visited  in 
the  STRES  program  (See  Table  5).  In  addition,  the  Air  Force  Advanced 
Simulator  for  Pilot  Training  (ASPT)  research  device  has  an  auto-demo 
capability. 

Utility  Information 

User  acceptance  and  frequency  of  use  of  auto-demo  were  found  to  vary 
significantly  among  the  operational  training  sites  visited  during  the 
program.  Over  the  course  of  the  visits,  however,  a  pattern  emerged  In 
terms  of  training  contexts  and  student  experience  and  skill  levels  that 
were  associated  with  more  frequent  use  and  acceptance  of  the  auto-demo 
feature.  The  observations  can  be  summarized  as  follows: 

The  training  value  of  an  automated  demonstration  will  be  greatest 
when  the  cues,  responses  and  task  performance  requirements  being 
demonstrated  are  new  to  the  student  (i.e.,  they  are  not  highly 
familiar  to  him). 

The  above  statement  is  based  on  observations  of  uses  of  fixed  wing 
ATDs  listed  In  Table  4,  and  discussions  with  the  Instructors  and 
students  using  these  devices.  Another  way  of  stating  this  is  that  the 
value  of  the  auto-demo  feature  appears  to  lie  In  applications  which 
Involve  Introducing  aircrew  members  to  an  aircraft  they  have  not  flown 
before,  or  to  applications  of  the  aircraft  and/or  Its  systems  that  they 
have  not  been  exposed  to  before.  Tht*s,  it  is  a  fair  estimate  that  the 
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Table  5.  Surveyed  ATDs  Equipped  With  Auto-Demo  Feature 


Type  of  Training 

Ai rcraft 

Purpose 

Undergraduate 

Pilot  Training 

T-37 

Introduction  of  UPT 
students  to  selected 
maneuvers.  Typically 
supplemented  by  verbal 
comnentary  from  the 
Instructor  In  the 
cockpit. 

UH-1 

Introduction  to.  and 
coaching  during 
instrument  takeoff 
training.  Audio  tape 
played  to  provide 
Instructor  commentary. 

Transition  S 
Continuation 

Training 

HH-52 

H-3 

Introduction  to 
Instrument  takeoff 
procedures.  Used  to 
demonstrate  the  precise 
control  required  for 
this  maneuver  In  these 
aircraft.  Used  at 
Instructor's  discretion 
to  Introduce  auto 
rotation  procedures. 

A-7E 

Demonstration  of  cues, 
technique  and 
procedures  for  night 
carrier  landings. 

C-5 

DC-10 

Feature  available  on 
these  devices,  but  not 
In  use  during  the 
period  covered  by  this 
program. 
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value  of  auto-demo  will  be  greatest  for  undergraduate  and  transition 
training. 

The  above  conclusion  is  based  on  the  site  visit  evidence  that 
auto-demo  is  used  largely  where  new  pilots  are  being  trained,  and  where 
relatively  new  pilots  are  being  transitioned  for  their  initial 
operational  assignment.  Related  to  this  was  the  observation  that 
auto-demo  never  was  used  when  seasoned  pilots  were  transitioned  to 
different  aircraft  where  mission  and  task  requirements  (and  cue,  task 
and  performance  requirements)  were  highly  similar  to  those  in  their 
immediately  prior  assignment. 

Related  Research 


There  is  a  general  lack  of  empirical  research  or  even  controlled 
observations  directly  concerned  with  the  instructional  efficiency  or 
effectiveness  of  virtually  any  Instructional  support  features.  This  is 
the  case  with  auto-demo,  although  three  studies  were  found  that  dealt 
with  demonstrations. 

Hughes,  Hannan  and  Jones  (1979)  have  reported  the  only  known 
experimental  study  of  the  use  of  auto-demo  and  record/replay 
instructional  features.  Their  study  was  designed  to  investigate 
implications  of  the  use  of  auto-demo  and  record/replay  features  in  the 
context  of  automated  instructional  approaches  to  flying  training. 
Fifteen  Air  Force  UPT  students  were  divided  into  three  groups.  Training 
was  conducted  in  the  Air  Force  Advanced  Simulator  for  Pilot  Training 
(ASPT),  which  simulated  the  T-37  aircraft.  Each  student  pilot's  task 
was  to  fly  the  first  two  leaves  of  a  cloverleaf  in  the  visually  equipped 
simulator.  None  had  flown  the  maneuver  before. 

All  of  the  student  pilots  received  a  videotaped  introduction  to  the 
study,  which  Included  a  five  minute  preflight  briefing  on  how  to  perform 
the  cloverleaf.  Each  also  received  an  automated  demonstration  of  proper 
performance  of  the  first  two  leaves  of  the  cloverleaf  maneuver  in  the 
simulator.  Subsequent  training  then  differed  for  the  three  groups  of 
student  pilots.  One  group  flew  two  leaves  of  the  cloverleaf  for  four 
blocks  of  three  attempts  each.  This  group  received  an  auto-demo  of 
correct  maneuver  performance  following  each  block  of  three  attempts. 
Similarly,  a  second  group  received  a  replay  of  their  most  recent  attempt 
at  the  maneuver  following  each  block  of  three  attempts.  A  third  group 
(the  control  group)  was  given  an  additional  attempt  at  the  maneuver 
after  each  block  of  three  trials  In  lieu  of  either  a  replay  or  a  demo. 

The  performance  of  all  of  the  student  pilots  was  scored  In  two  ways. 
An  Instructor  was  located  in  a  second,  "slaved"  simulator  cockpit,  which 
provided  all  Instrument  readings  and  out-of-cockpit  visual  cues 
available  to  the  students.  After  each  execution  of  each  maneuver  for 
students  In  all  three  groups,  the  Instructor  rated  the  maneuver  as: 
unsat Isfactory;  fair;  good;  or  excellent.  Each  student  was  told  his 
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rating  after  each  maneuver.  In  addition,  maneuver  performance  was 
scored  using  an  automated  performance  measurement  system. 

Results  of  the  study  by  Hughes,  et  al.  showed  that  the  group  that 
received  additional  practice  instead  of  either  auto-demo  or  replay  did 
the  best.  The  group  that  received  a  replay  of  their  last  maneuver 
attempt  did  second  best,  and  the  group  that  received  the  correct 

performance  demonstration  came  in  third.  Remembering  that  students  in 
each  group  received  an  instructor  rating  of  their  performance  after  each 
try  at  the  maneuver,  the  findings  at  first  suggest  that  auto-demo  and 
record/replay  may  have  little  instructional  value.  However,  both  of 
these  instructional  features  were  used  in  the  context  of  "automated 
instruction"  in  which  detailed  instructor  evaluation,  guidance  and  other 
interaction  were  excluded.  What  the  findings  do  show  is  that  in  this 
context,  the  features  investigated  showed  little  promise.  What  future 
research  should  address  is  the  more  operationally  oriented  use  of 
auto-demo  and  record/ replay,  where  the  features  are  used  at  the 

instructor's  discretion  to  correct  specific  performance  deficiencies  and 
where  instructor-student  interaction  is  tailored  to  student  learning 
needs.  It  also  must  be  noted  that  other  research  has  indicated  that  the 
ASPT  simulation  may  not  be  well  suited  for  training  aerobatic  maneuvers. 
(See  Fidelity  Volume,  Chapter  III) 

In  the  context  of  Navy  research  with  the  A- 7  Night  Carrier  Landing 
Trainer  (NCLT),  it  has  been  shown  that  auto-demo  has  a  significant,  but 
non-quantifiable  instructional  value  (Brictson  and  Burger,  1976).  In 
the  study,  instructors  used  auto-demo  as  an  integral  part  of  NCLT 

training.  However,  there  are  no  data  quantifying  the  specific 

contribution  that  the  feature  made  to  overall  training  effectiveness  or 
efficiency.  This  application  does  represent  a  case,  however,  where  an 
Instructor-flown  demonstration  was  not  possible  because  the  NCLT  ATD  was 
a  single  seat  configuration.  No  other  method  for  demonstration  was 
possible. 

In  the  Army's  Nap  Interpretation  and  Terrain  Analysis  Course 
(MITAC),  both  undergraduate  and  experienced  pilots  are  presented  with 
the  problem  of  recognizing  and  correlating  map  features  with  real  world 
terrain  features  during  a  series  of  film  based  exercises.  This  training 
is  presented  to  the  Individual  aircrew  member,  and  the  pacing  of  the 
exercise  Is  controlled  to  match  either  the  level  of  instruction  or  the 
speed  at  which  the  student  would  experience  the  cues  In  the  real  world 
(a  function  of  aircraft  speed).  This  course  has  proved  quite 
successful,  and  represents  the  application  of  this  feature  In  a  simple 
ATD  to  solve  a  tactical  training  problem. 

In  sunmary,  it  must  be  concluded  that  additional,  operationally 
oriented  research  on  auto-demo  and  virtually  all  other  Instructional 
support  features  is  sorely  needed. 
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Design  Considerations 

There  are  two  basic  methods  for  creating  auto-demos.  One  is  to  use 
an  ATD's  record  and  replay  capability.  This  method  requires  that  a 
proficient  aircrew  member  fly  the  necessary  profiles  in  the  desired 
manner  using  desired  techniques.  This  method  has  the  advantage  that 
highly  proficient  crewmembers  usually  are  available.  It  has  the 
disadvantage  that  it  is  difficult,  even  for  highly  proficient  aircrew 
members,  to  fly  profiles  with  the  perfection  they  judge  necessary  to 
provide  a  performance  model  for  the  student  to  pattern  after.  Also,  the 
lack  of  well  defined  criteria  for  the  largest  percentage  of  flying 
training  tasks  makes  it  difficult  for  Instructors  to  agree  on  what 
constitutes  acceptable  performance.  Peer  pressure  also  plays  a  role  in 
that  no  pilot  wishes  to  have  his  "mistakes"  recorded  for  others  to 
observe.  Consequently,  acceptable  demonstrations  have  proved  difficult 
to  develop.  Finally,  since  "no  one"  flies  "perfect"  maneuvers  in  the 
real  world,  the  need  to  demonstrate  perfection  (versus  typical 
performance)  must  be  questioned. 

A  procedure  often  used  is  to  draw  on  the  joint  capabilities  of 

replay,  freeze  and  fly-out  (where  the  pilot  at  the  controls  takes  over 
after  a  segment  of  canned  maneuvering).  The  procedure  requires  that  the 
initial  segnent  of  the  maneuver  for  which  the  demonstration  is  that 

being  developed  be  flown  and  recorded.  The  ATD  is  frozen  at  the  point 
where  desired  performance  is  not  being  achieved  by  the  pilot  making  the 
demonstration.  The  ATD  is  reinitialized  to  the  beginning  of  the 

maneuver  segment.  The  segment  is  then  replayed  to  a  point  just  prior  to 
where  performance  was  judged  unacceptable.  The  pilot  then  takes  over 
control  manually  (i.e.,  flies-out)  and  attempts  additional  segments  of 
the  maneuver.  The  procedure  is  repeated  until  the  entire  mission 

segment  or  maneuver  is  judged  acceptable  by  the  pilot.  The  maneuver 
then  is  stored  and  made  accessible  for  subsequent  call  up  as  an 
automated  demonstration. 

The  second  method  for  creating  auto-demos  is  through  the  development 
of  customized  computer  software  for  each  demonstration.  This  approach 
has  the  advantage  that  system  analysts  and  computer  programmers  probably 
can  develop  and  implement  mathematically  perfect  demonstrations.  The 
method  has  the  distinct  disadvantage,  however,  of  requiring  computer 
software  specialists  to  create  new  demonstrations  or  to  modify  initial 
demonstrations  following  delivery  of  the  ATD.  Since  highly  specialized 
skills  are  required,  development  of  automated  demonstrations  using  this 
method  would  require  either  the  creation  of  a  new  simulator  support 
billet  or  the  need  to  rely  on  contract  support  services.  This  requires 
substantial  agreement  and  cooperation  between  the  ATD  developer  and  the 
user  community  with  respect  to  identifying  and  collecting  the 
performance  data  required  to  model  desired  performance.  Both 
manufacturer  and  user  personnel  view  the  direct  software  approach  as  the 
more  time  consuming  and  costly  of  the  two  methods  available.  Again,  the 
need  for  "perfect"  demonstrations  must  be  questioned. 
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Once  created,  the  spectrum  of  auto-demos  must  be  clearly  identified 
to  and  accessible  by  the  ATD  instructor/operator. 

The  number  of  auto-demos  and  the  duration  of  each  also  is  a  design 
consideration.  Each  of  these  factors  must  be  addressed  with  respect  to 
specific  ATD  and  training  program  applications.  General  rules  of  thumb 
do  not  apply. 

Incorporating  fully  synchronized  voice  audio  into  an  auto-demo 
capability  should  be  considered  from  three  standpoints.  One,  it 
provides  a  means  to  present  highly  standardized  and  complete 
instructional  information.  Second,  it  provides  verbal  instructional 
content  to  students  who  may  be  using  an  ATD  for  "extra  time"  training 
when  a  qualified  instructor  may  not  be  present.  Third,  it  may  provide 
the  only  means  for  fully  coordinated  instructional  commentary  in  cases 
where  the  instructor  cannot  meaningfully  observe  what  the  student  is 
observing  and  doing.  An  example  would  be  a  single  place  ATD  with  an  out 
of  the  cockpit  visual  system,  and  a  less  than  adequate  presentation 
(repeater)  of  the  visual  scene  at  the  instructor/operator  console. 

Hughes  (1979)  correctly  observes  that  present  uses  of  automated 
demonstrations  are  highly  traditional  and  rudimentary,  where  the  student 
sits  and  passively  watches  a  canned  execution  of  the  task.  Almost 
Invariably,  the  canned  execution  represents  "perfect"  or  text  book 
performance.  Hughes  also  notes  that  significant  research  issues  remain, 
for  example,  in  determining  the  content  and  use  of  automated 
demonstrations.  At  issue  is  the  need  to  consider  innovative  content  and 
uses  of  auto-demo  and  to  develop  sound  relationships  of  this  feature  to 
others.  For  example,  consideration  should  be  given  to  enabling  the  use 
of  total  system  freeze  during  demonstrations.  Where  this  capability  has 
existed,  it  has  been  used.  Similarly,  enabling  the  use  of  ground 
position  freeze,  parameter  freeze  and  axis  lock  during  the  creation  of 
demonstrations  may  make  their  development  easier. 

The  capability  to  terminate  an  ongoing  demonstration  and 
re-initialize  the  ATD  also  must  be  considered  from  the  standpoint  of 
making  the  most  efficient  use  of  training  time  and  In  terms  of 
Instructor/student  acceptance  of  the  feature.  Similarly,  the  capability 
to  stop  an  automated  demonstration  and  re-initiate  It  from  the  beginning 
must  be  considered,  especially  for  fairly  lengthy  demonstrations  (e.g., 
longer  than  one  or  two  minutes).  This  provides  the  ability  to  "go  back 
and  look  at  It  again"  without  being  committed  to  waiting  out  the  balance 
of  the  demonstration.  Along  the  same  line,  it  should  be  possible  to 
enter  an  auto-demo  at  desired  points  so  that  it  is  not  necessary  to 
"wait  for"  the  needed  element  of  the  auto-demo.  Both  design  factors  are 
needed  for  user  acceptance  and  use. 

From  a  content  standpoint,  possible  benefits  of  demonstrating 
typical  student  errors  and  consequences  needs  more  systematic  attention. 
Similarly,  possible  benefits  of  demonstrating  alternative  solutions  to  a 
flight  or  tactical  problem  need  to  be  addressed  to  promote  coping  with 
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unexpected  conditions.  Either  of  these  might  also  incorporate  the  use 
of  augmenting  cues,  such  as  a  flight  path  predicator  or  a  weapon  impact 
predictor  information  presented  by  means  of  an  out  of  the  cockpit  visual 
display  system. 

RECORD  AND  REPLAY 


Summary 

«  • 

Record  and  replay  makes  it  possible  to  recreate  a  segment  of  flight 
just  as  it  was  flown,  without  direct  intervention  by  the  student  or  the 
instructor/operator.  The  last  five  consecutive  minutes  of  flight  can  be 
preserved  for  replay  in  most  current  ATDs.  It  commonly  is  assumed 
that  record  and  replay  provide  a  highly  detailed  and  dynamic  memory  aid. 
The  ability  to  freeze  the  ATD  during  replay  is  clearly  a  desired  feature 
for  most  training  applications.  The  ability  to  "fly  out"  of  replay  also 
is  desirable.  Application  of  the  record  and  replay  feature  has  been 
limited  to  more  sophisticated  ATDs  (e.g.  OFTs,  PTTs,  and  FMTs).  Student 
proficiency  appears  to  be  the  primary  factor  In  deciding  whether  to 
incorporate  record  and  replay  into  an  ATD.  It  appears  that  the  greatest 
training  value  for  record  and  replay  is  in  cases  where  the  student  is 
not  familiar  with  the  specific  task  (cues  and  responses)  he  is  learning, 
and  where  task  loading,  timesharing  and  pacing  are  problems. 

Description 

Record  and  replay  is  the  capability  of  an  ATD  to  record  relevant 
system  parameters  and  then  use  these  data  to  present  student  pilot 
performance  in  a  review  mode.  In  a  typical  installation,  the  last  five 
continuous  minutes  of  performance  are  recorded.  After  freezing  the  ATD, 
typically  using  total  system  freeze,  the  Instructor  selects  the  point  in 
time  in  the  available  replay  period  that  he  wants  replay  to  be 
initiated.  Typically,  the  selection  Is  limited  to  one  minute 
increments.  When  replay  Is  initiated,  the  ATD  initializes  to  the 
conditions  prevailing  at  the  replay  start  point.  All  events  of 
consequence  are  reproduced  during  replay,  including  the  visual  scene  (if 
present),  motion  system  operation,  primary  flight  control  and  display 
movements,  and  appropriate  sensor  display  content.  The  ATD  performs  as 
though  the  student  was  again  flying  the  replayed  segment  exactly  as  It 
had  been  flown  before.  Technically,  replay  can  be  accomplished  for  any^ 
segment  of  flight.  Replay  of  voice  communications  is  possible,  but  very 
few  existing  devices  incorporate  the  capability  due  to  technical 
difficulties  which  have  resulted  In  a  lack  of  synchronization  between 
verbal  message  content  and  system  performance.  This  has  been  a  handicap 
because  relevant  cues  are  not  correlated.  Modern  digital  voice  storage, 
however,  offers  a  solution  to  this  problem. 

Replay  can  be  terminated  In  one  of  three  ways.  First,  the 
Instructor  can  freeze  the  ATD  and  select  a  different  mode  of  operation. 
Second,  replay  can  be  de-selected,  and  the  student  can  resume  manual 
control  (i.e.,  fly  out  of  replay).  Third,  replay  can  be  allowed  to 


continue  throughout  the  period  for  which  replay  data  had  been  recorded. 
The  ATD  can  be  frozen  or  manual  control  assumed  following  the  completion 
of  replay. 

The  record  and  replay  feature  requires  substantial  two-way 
comnunication  between  ATD  controlling  software  and  hardware,  and  crew 
station  displays  and  controls.  Thus,  the  record  and  replay  feature  is 
found  only  in  more  sophisticated,  computer  controlled  ATDs.  ATD 
hardware  and  software  must  be  configured  to  induce  changes  in  the 
operation  of  simulated  subsystems  without  any  intervention  by  the 
student  or  the  instructor. 

Related  Support  Features 

The  following  paragraphs  identify  relationships  to  other 
instructional  features. 

Automated  Demonstration.  Automated  demonstrations  permit  an  ATD  to 
reproduce  a  previously  recorded  segment  or  phase  of  flight  without 
direct  intervention  by  the  student  or  the  instructor.  '  Record  and 
replay  are  used  in  some  ATDs  to  create  demonstrations.  (See  the 
auto-demo  section  of  this  chapter). 

Freeze.  The  freeze  feature  allows  the  instructor  to  suspend  some  or 
all  device  parameters  associated  with  the  student's  task.  This  can 
range  from  suspending  all  parameters  (total  system  freeze)  through 
suspending  only  selected  parameters  (parameter  freeze).  .(See  the 
freeze  section  of  this  chapter). 

Where  the  record  and  replay  feature  was  observed,  total  system 
freeze  was  used  to  stop  the  simulation  prior  to  entering  the  replay 
mode  and,  on  many  occasions,  to  temporarily  interrupt  replay  or  to 
terminate  replay. 

Svnthesi zed  Control  1 er .  Synthesized  speech  (i.e.  computer  generated 
speech)  is  a  relatively  new  instructional  support  technology  that, 
to  date,  has  been  incorporated  in  only  a  few  ATDs.  If  such  a 
capability  were  Incorporated,  for  example  an  automated  GCA 
controller,  it  is  logical  that  such  controllers  should  operate 
during  replay  since  they  provide  guidance  Information  used  by  the 
students-  (See  Chapter  VI  of  this  report). 

Instructional  Values 

Historically,  replay  has  been  most  often  used  when  a  student  has  a 
performance  problem  and  It  Is  the  Instructor's  judgement  that  there  Is 
little  or  no  training  value  In  continuing  practice  of  the  maneuver 
without  personal  intervention.  The  commonly  assumed  instructional  value 
of  record  and  replay  Is  that  It  provides  a  highly  detailed  and  dynamic 
memory  aid.  A  directly  related  value  Is  that  student  task  loading  is 
considerably  reduced  during  replay.  This  provides  a  better  environment 
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to  work  with  the  instructor  to  review  performance,  identify  performance 
problems  and  resulting  errors,  analyze  causes  underlying  performance 
problems,  and  receive  guidance.  Total  system  freeze  is  a  directly 
related  feature  that  allows  replay  to  be  temporarily  interrupted  for 
instructional  interaction. 

A  second  value  for  record  and  replay  often  cited  by  instructors  was 
the  use  of  replay  to  resolve  disagreements  between  the  instructor  and 
the  student.  This  use  is  valuable  in  cases  where  the  student  is 
temporarily  overloaded  or  distracted  and  may  not  recognize  that  he  has  a 
performance  deficiency.  In  this  role,  replay  also  can  enhance  the 
instructor's  credibility  as  a  positive  and  motivated  aircrew  trainer. 

A  review  of  empirically  supported  principles  of  learning  indicates 
that  the  primary  instructional  values  of  record  and  replay  reside  in 
the  following  instructional  principles: 

Control  cues  to  make  sure  that  the  student  is  forming  proper 

associations,  i.e.,  responding  appropriately  to  correct  cues. 

Emphasize  differences  between  correct  and  incorrect  cue  forms. 

With  very  complex  tasks,  instruction  in  principles  yields  better 

results. than  laying  down  a  detailed  drill,  while  with  simpler  tasks 

drill  i?  at  least  equally  effective. 

A  number  of  instructors  who  were  interviewed  suggested  that 
replaying  performance  that  was  well  within  criteria  of  acceptance  can  be 
motivational  by  providing  positive  reinforcement  to  the  student  for  a 
job  well  done.  This  likely  over- statement  should  be  tested  empirically. 
A  majority  suggested  that  this  use  of  replay  should  follow  soon  after 
the  student  works  through  a  performance  problem  and  demonstrates 
acceptable  performance.  Others  felt  that  the  use  of  replay  should  not 
be  limited  to  this  use,  alone. 

Record  and  replay  was  used  at  the  sites  visited  almost  exclusively 
for  pilot  and/or  copilot  training.  It  is  quite  possible,  however,  that 
replay  could  be  used  in  a  crew  training  environment  as  a  means  to 
evaluate  and  improve  crew  interaction.  Voice  replay  also  would  be 
required  in  this  application  since  many  of  the  indicators  of  crew 
Interaction  hinge  on  crew  communications.  This  potential  need  for 
replay  depends  on  how  well  individual  crewmember  responsibilities  are 
defined,  how  students  are  trained  and  evaluated,  and  how  productive 
replay  would  be  in  terms  of  Interactive  crew  training. 

Only  one  device  was  surveyed  that  incorporated  a  slow  motion  replay 
capability.  The  slow  motion  replay  was  at  one  half  real  time.  The 
capability  was  not  used  because  it  was  "unrealistic".  Thus,  slow  motion 
replay  remains  an  untried  capability  for  all  practical  purposes. 
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Record  and  replay  also  holds  the  potential  of  providing  a  set  of 
exercises  that  can  be  used  to  guide  standardization  and  evaluation 
personnel  on  the  performance  that  is  expected  of  aircrew  members  during 
and  at  the  end  of  formalized  instruction.  A  similar  case  can  be  made 
for  continuation  training. 

Observed  Applications 

A  number  of  ATDs  observed  during  site  visits  incorporated  record  and 
replay.  All  were  OFTs  or  PTTs.  Training  applications  are  summarized  in 
Table  6.  Operational  experience  was  sampled  during  training  on  the 
following  tasks. 

Landings  and  Takeoffs:  Based  on  the  survey  of  operational  training 
sites,  record  and  replay  has  been  used  successfully  during  training 
for  terminal  area  operations.  These  tasks  include:  visual  takeoff; 
instrument  takeoff;  standard  departures;  instrument  approaches  (e.g. 
VOR);  instrument  final  approach,  visual  final  approach; 
autorotations;  and  night  carrier  landings.  The  record/replay 
feature  tended  to  be  used  more  during  undergraduate  and  early 
transition  training  because  students  were  least  familiar  with  their 
tasks  at  that  stage  of  their  training. 

Air-to-Air  Combat:  Record  and  replay  has  been  used  during  initial 
pilot  training  in  air  tactics  in  the  Simulator  for  Air  to  Air  Combat 
(SAAC).  Record  and  replay  reportedly  is  not  used  during  latter 
stages  of  advanced  training  (e.g.  TAC  ACES  Program),  where  pilots 
are  more  proficient  in  air  combat  maneuvering  and  tactics.  This 
follows  since  most  pilots  "know"  what  is  expected  and  simply  need 
practice  late  in  training. 

Utility  Information 

Two  trends  were  apparent  at  the  sites  visited  that  had  ATOs  with 
record  and  replay.  One  involved  the  use  of  record  and  replay  in 
relation  to  student  proficiency  level  and  knowledge  of  the  training 
tasks  to  be  performed.  The  .  second  Involved  instructor- to- instructor 
variability  on  how  often  and  when  they  used  record  and  replay. 


Interview  results  showed  that  record  and  replay  were  used  more  often 
during  training  on  tasks  that  were  new  to  the  student  and  were 
relatively  complex  for  Mm.  In  general,  the  use  of  record  and  replay 
centered  on  undergraduate  pilot  training  and  on  the  training  of  new  (to 
the  pilot)  and  relatively  complex  advanced  flying  skills,  including 
night  carrier  landing  and  air  to  air  combat  maneuvering.  On  the  other 
hand,  replay  was  not  used  during  transition  training  or  continuation 
training  where  pilots  had  a  basic  understanding  of  what  was  required, 
and  unless  there  was  no  argument  on  performance  deficiencies.  This 
trend  was  observed  during  military  and  commercial  airline  training.  The 
same  trend  was  found  for  the  use  of  automated  demonstrations. 


Table  6.  Surveyed  ATDs  Equipped  With  Record  and  Replay  Feature 


Type  of  Training 

Aircraft  Simulated 

Purpose 

Undergraduate 

Pilot  Training 

T-37/T-38 

Used  in  initial  pilot 
training  of  various 
instrument  flight 
procedures,  and  visual 
take  off  and  landing. 

Transition  & 
Continuation 

Training 

F-4E  (SAAC) 

Used  to  demonstrate 
performance  problems 
and  diagnose  causes 
for  pilots  who  are  just 
learning  air  combat 
maneuvering. 

A-7E  (NCLT) 

Used  to  demonstrate 
(as  Auto  Demo)  cues, 
techniques  and  pro¬ 
cedures  during  night 
carrier  landings.  Used 
to  diagnose  performance 
problems.  Also  used  to 
reinforce  well  executed 
approaches  and 
recoveries. 

HH-52 

H-3 

Used  to  demonstrate 
performance  problems, 
diagnose  causes  and 
reinforce  correct 
executions  of 
instrument  takeoffs, 
landings  and  auto- 
rotations. 

UH-1 

CH-47 

Same  as  HH-52  and  H-3 
above,  plus  terminal 
area  instrument 
procedures. 
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Table  6.  (Continued) 

\ 

■  Type  of  Training 

Aircraft  Simulated 

Purpose 

I 

C-5 

Feature  Available  on 

1 

DC-10 

these  devices  but  not 

B-747 

used  during  the  period 
covered  by  this 
program.  (Devices  were 
used  by  highly 
experienced  pilots.) 

E-3A  (OFT) 

/ 

Capability  Exists,  but 
not  used  due  to  hard¬ 
ware  problems. 

‘ 

E-3A  (MT) 

Capability  exists  for 
replaying  portions  of 
tactical  mission 
exercises. 

Research  & 

ASPT 

Diagnosis  and  pre- 

1  Development 

scription  for  pilot 
subjects  participating 

In  training  research 

programs .  , 

Instructors  were  quite  consistent  In  their  explanations.  Replay  Is 
most  useful  when  the  student  Is  not  aware  that  he  had  made  an  error  or 
Is  uncertain  of  the  precise  cause  of  his  error  and  resulting  performance 
problem.  This  situation  was  most  apt  to  arise  when  tasks  were  being 
Initially  learned.  For  example,  relevant  cues  may  be  missed;  students 
may  become  disoriented;  they  may  not  be  able  to  correctly  Interpret 
displays  or  visual  cues;  they  may  not  understand  flight  control, 
navigation  or  system  operation  principles  and  procedures.  In  such 
cases,  replay  was  accepted  and  used  as  a  meaningful  Instructional  tool. 
On  the  other  hand,  pilots  who  are  more  highly  experienced  and  skilled  In 
the  tasks  to  be  performed  generally  are  able  to  Identify  their  own 
performance  deficiencies,  diagnose  the  causes,  and  take  corrective  steps 
without  the  need  for  a  highly  detailed  memory  and  diagnostic  aid  such  as 
record  and  replay. 

The  following  principle  of  use  applies  for  record  and  replay: 

The  training  value  of  record  and  replay  will  be  greatest  when  the 
cues,  responses  and  task  performance  requirements  being  learned  are 
new  to  the  student. 

All  of  the  record  and  replay  capabilities  surveyed  could  replay  up 
to  the  last  five  consecutive  minutes  of  flight.  The  portion  to  be 
replayed  could  be  selected  In  one  minute  Increments.  The  total  five 
minute  span  was  judged  to  be  quite  adequate.  In  fact,  no  one  was 
interviewed  who  had  replayed  anything  approaching  five  minutes  of 
flight.  A  frequent  comnent  was  that  one  minute  increments  of  replay 
often  were  excessive,  and  that  shorter  time  increments  should  be 
considered  In  the  future.  The  F-16  ATO  will  allow  replay  to  be  selected 
in  ten  second  increments. 

Related  Research 


Only  one  research  study  was  found  that  investigated  the  use  of 
record  and  replay  (Hughes,  Hannan  and  Jones,  1979).  The  study  Is 
summarized  here;  details  are  presented  in  the  section  of  this  chapter 
dealing  with  automated  demonstrations. 

Hughes  et  al.  had  three  groups  of  Air  Force  UPT  student  pilots  learn 
to  fly  the  first  two  leaves  of  the  cl  overleaf  maneuver  In  a  visually 
equipped  simulation  of  the  T-37  aircraft.  One  group  of  students  viewed 
an  automated  demonstration  of  the  maneuver  being  performed  correctly 
after  each  block  of  three  attempts  at  the  maneuver.  A  second  group 
received  a  replay  of  their  last  attempt  after  each  block  of  three 
trials.  A  third  group  received  one  additional  trial  at  performing  the 
maneuver  following  each  block  of  three  attempts  In  lieu  of  either  a 
demonstration  or  a  replay.  Performance  was  scored  by  an  Instructor  In  a 
second,  slaved  simulator;  performance  also  was  measured  using  an 
automated  performance  measurement  system. 
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Results  of  the  study  showed  that  the  group  that  received  additional 
practice  instead  of  either  the  automated  demonstration  or  replay  did  the 
best.  The  group  that  received  a  replay  of  their  last  maneuver  attempt 
did  second  best,  and  the  group  that  received  the  correct  performance 
demonstration  came  in  third.  The  findings  at  first  suggest  that  the 
automated  demonstration  and  record/replay  features  may  have  little 
instructional  value.  However,  both  of  these  instructional  features  were 
used  in  the  context  of  "automated  instruction"  in  which  detailed 
instructor  evaluation,  guidance  or  other  interaction  was  excluded.  What 
the  findings  do  show  is  that  in  this  context,  the  features  investigated 
showed  little  promise.  What  future  research  should  address  is  the  more 
operationally  oriented  use  of  automated  demonstrations  and 
record/replay,  where  the  features  are  used  at  the  instructor's 
discretion  to  correct  specific  performance  deficiencies  and  where 
student- instructor  interaction  is  tailored  to  student  learning  needs. 

Design  Considerations 

This  section  addresses  five  record  and  replay  design  considerations. 
The  first  is  the  number  of  minutes  of  flight  that  needs  to  be  recorded. 
Hard  and  fast  figures  cannot  be  given.  For  the  applications 
investigated,  a  total  of  five  minutes  was  more  then  adequate. 
Instructor  consensus  was  that  for  most  uses,  1.5  to  2.0  minutes  probably 
would  be  sufficient.  However,  the  amount  of  replay  time  to  be 
incorporated  into  an  ATD  must  be  determined  on  a  case  by  case  basis.  It 
may  be  that  for  application  to  crew  coordination  situations  a  longer 
replay  capability  is  appropriate. 

The  second  design  consideration  is  how  best  to  select  the  point  at 
which  replay  is  to  be  initiated.  The  systems  surveyed  all  required  the 
instructor  to  select  replay  in  one  minute  increments.  Thus,  the  last 
one,  two,  etc.  minutes  (up  to  five)  could  be  replayed.  A  number  of 
instructors  commented,  however,  that  one  minute  "is  a  long  time  to 
replay".  Some  suggested  selecting  replay  in  30  second  increments.  The 
F-16  ATD  provides  for  10  second  increments.  Identifying  the  "best" 
increment  requires  further  research  and  operational  experience. 

An  alternative  to  specific  time  increments  also  should  be  examined. 
A  reasonable  alternative  is  to  provide  the  instructor  with  a  cuing 
control.  Replay  then  could  be  started  at  a  determined  time  prior  to 
when  the  cuing  control  was  activated  (e.g.  20  seconds,  or  whatever  is 
reasonable  for  the  particular  training  application).  This  would  allow 
instructors  to  relate  replay  to  training  events  rather  than  clock  time. 

Third,  voice  replay  obviously  should  receive  serious  consideration 
for  training  tasks  where  voice  conmunl cation  is  a  significant  element  of 
total  task  performance.  Examples  Include  controller  messages  and  crew 
or  team  performance.  Historically,  problems  have  existed  in  rapidly 
initializing  to  the  proper  point  in  recorded  voice  messages,  and  in 
keeping  message  content  synchronized  during  replay.  These  problems 
have  stemned  from  using  magnetic  tape  technology  for  storing  the 
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messages.  Digital  voice  recording  technology  now  is  available  to 
synchronize  voice  message  content  with  other  cues  during  replay, 
particularly  vision  and  motion  cues. 

The  fourth  consideration  involves  the  use  of  freeze  together  with 
record  and  replay.  Total  system  freeze  is  necessary  to  stop  the 
simulation  while  replay  is  selected.  Similarly,  total  system  freeze  is 
necessary  to  stop  replay  in  order  to  reinitialize  the  ATD. 

There  may  be  training  and  user  acceptance  values  in  being  able  to 
deactivate  replay  part  way  through,  thereby  allowing  the  student  to 
assume  manual  control  during  a  "fly  out"  capability.  It  is  suggested 
that  the  capability  be  designed  into  ATDs  to  provide  research 
capabilities  as  well  as  potentially  enhanced  training  value. 

MANUAL  AND  PROGRAMMABLE  INITIALIZATION 

Summary 

Every  operational  flight  trainer  and  full  mission  trainer  surveyed 
had  a  capability  for  Initializing/reinitializing  aspects  of  the 
simulation.  The  objective  of  initializing  is  to  position  and  to  some 
extent  configure  the  ATD  with  econony  of  time  and  effort  to  support 
training.  Modern  ATDs  using  digital  computer  technology  can  be 
initialized  easily.  Other,  older  ATDs  sometimes  require  lengthy  manual 
procedures,  such  as  slewing.  In  all  cases,  initializing  is  geared  to 
initially  setting  up  the  simulation  for  training,  or  to  establishing 
meaningful  conditions  (reinitializing)  for  specialized  task  training 
during  an  ATD  session.  The  feature  is  central  to  efficient  ATD 
utilization. 

Feature  Description 

One  of  the  greatest  assets  of  ATDs  is  that  time  need  not  be  used 
unproductlvely  on  events  of  no  Imnedlate  training  value.  For  example. 
In  an  ATD  it  is  not  necessary  to  fly  a  missed  approach  profile  just  to 
practice  final  approach  and  landing.  Similarly,  it  is  not  necessary  to 
fly  the  approach  and  stabilization  precontact  segments  of  air  refueling 
to  practice  hook-up  and  breakaway.  Instead,  ATDs  can  be  almost 
instantaneously  Initialized  to  desired  conditions  for  the  start  of 
training. 

Initialization  Involves  specifying,  usually  from  the 
instructor/operator  console,  the  parameters  of  Interest  and  their 
values.  Parameters  to  be  initialized  vary  with  the  training 
applications  of  the  ATD.  Parameters  most  conmonly  Initialized  are  those 
that  determine  the  ATD's  simulated  position  with  respect  to  the  earth. 
These  Include:  position  (e.g.  latitude  and  longitude);  altitude; 
heading;  speed;  and  attitude.  Other  factors  such  as  weapons  load, 
amnunltlon  and  fuel  load  also  can  be  initialized.  ATDs  with  out  of 
cockpit  visual  systems  have  capabilities  for  Initialization  of  specific 
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visual  scene  characteristics.  These  Include:  day,  dusk  or  night  scene; 
ceiling;  broken  or  complete  overcast;  runway  visual  range;  amount  of 
simulated  haze  near  the  ground;  Intensity  of  approach,  runway  and  other 
simulated  lighting;  and  simulated  weapon  visual  effects,  such  as  surface 
to  air  missile  launches.  ATDs  used  for  weapon  delivery  training  also 
may  include  capabilities  to  initialize  the  weapons  carried, 
distribution,  and  simulated  mounting  points.  Threat  types,  locations 
and  other  characteristics  can  also  be  inltalized. 

Related  Instructional  Support  Features 

Total  system  freeze  was  used  In  each  ATD  surveyed  to  stop  the 
simulation  during  the  Inltlallzatlon/relnltiallzatlon  process.  When 
reinitialization  was  complete,  the  ATD  was  unfrozen  so  that  simulated 
flight  could  be  undertaken.  Total  system  freeze  was  discussed  earlier 
in  this  chapter. 

Instructional  Values 

Being  able  to  quickly  Initial ize/reinltial ize  ATDs  promotes 
efficient  use  of  available  training  time  and  other  assets.  A  good 
initialization  capability  minimizes  wasted  training  time  by  enabling 
rapid  set-up  of  training  conditions.  Time  is  not  required  to  "fly"  the 
ATD  to  a  desired  location,  for  example,  as  would  be  required  in  the  air. 
Rapid  initialization  is  one  of  the  distinguishing  features  of  ATDs. 

Observed  Applications 

Every  operational  flight  trainer  and  full  mission  trainer  surveyed 
during  the  program  had  some  capability  to  rapidly  initialize  the 
simulation.  The  pattern  of  use  of  the  feature  was  in  direct  keeping 
with  its  assumed  Instructional  support  value,  l.e.  to  make  the  most 
productive  use  of  available  training  time.  Undergraduate  and  transition 
training  programs  used  the  capability  to  provide  for  repeated  practice 
on  more  complex  tasks,  such  as  takeoff  and  final  approach  and  landing. 
Transport  category  programs  used  the  feature  to  "skip  over"  unproductive 
cruise  phases  of  missions.  Air  combat  simulations  (including  one 
observed  during  a  prior  research  program)  used  Initialization  to  quickly 
establish  Initial  conditions  for  attacker  and  adversary  aircraft  after 
each  engagement. 

Utility  Information 

ATDs  built  more  than  five  years  ago  tended  to  Incorporate  dedicated 
controls  for  Initializing.  Some  required  the  operation  of  several 
different  controls.  Including:  slewing  controls  to  reposition  ATD; 
and  separate  control  Inputs  for  desired  altitude,  speed,  etc.  Some 
required  that  the  desired  ATD  heading  be  established  In  the  ATO  cockpit 
using  primary  flight  controls  prior  to  setting  up  the  Initialization. 


Newer  devices,  and  several  older  ones.  Incorporated  dedicated 
controls  (usually  pushbuttons)  that  Instructors/operators  use  to  select 
one  of  several  programmed  combinations  or  sets  of  initial  conditions. 
These  controls  can  be  reprogrammed  off  line. 

Several  modern  ATDs  provided  instructors/operators  with  lists  of 
combinations  of  initialization  options  displayed  on  console  CRT 
displays.  The  desired  combination  is  selected  by  making  a  keyboard 
entry.  This  approach  is  much  like  the  dedicated  controls  approach,  but 
provides  two  added  advantages.  One  advantage  is  that  the  values  of  the 
initialization  parameters  can  be  spelled  out  on  the  CRT  display  as  a 
memory  aid  to  the  instructor.  The  second  is  that  more  option 
combinations  usually  can  be  made  available.  The  initialization  options 
can  be  reprogramed  off  line. 

Instructor/operator  consoles  using  CRT  display  and  keyboard  entry 
systems  also  can  display  each  Initialization  parameter,  and  allow  the 
instructor  to  Input  values  for  each  parameter.  This  approach  allows 
them  to  create,  on  line,  new  initialization  options.  Most  systems 
designed  this  way  save  the  initialization  values  until  they  are  changed 
by  the  Instructor.  This  allows  instructors  to  reinitialize  to  their 
specially  constructed  set  of  conditions  with  convenience,  because  they 
are  not  required  to  recreate  the  conditions  for  each  initialization. 

Related  Research 


No  research  was  found  dealing  specifically  with  alternative  designs 
or  uses  of  the  ATD  Initialization  feature. 

Design  Considerations 


A  careful  analysis  of  ATD  training  applications  must  precede 
attempts  to  identify  which  parameters  are  required  for  initialization, 
which  sets  of  parameters  are  to  be  combined  into  Initialization  option 
packages,  and  what  the  ranges  and  increments  of  parameter  values  should 
be.  General  rules  of  thumb  cannot  prevail.  This  follows  because 
Initial  conditions  and  condition  sets  must  be  geared  to  training 
requirements,  which  vary  among  training  programs. 

Maqy  future  ATD  Instructor/operator  consoles.  Including  retrofits, 
will  Incorporate  electronic  display  technology,  together  with  keyboard 
or  touch  panel  data  entry  systems.  These  technologies  are  well  suited 
for  displaying  programed  initialization  options  to  the  Instructor,  and 
allowing  him  to  select  given  options  or  create  new  ones  appropriate  to 
the  training  objectives  being  addressed.  In  addition  to  providing 
Instructors  with  Initialization  option  packages  to  select  from.  It  also 
Is  Instructlonally  desirable  to  enable  them  to  create,  at  their 
consoles,  additional  sets  of  Initial  conditions,  and  to  provide  the 
capability  for  storing  them  throughout  the  training  session  so  they  can 
be  recalled  for  repeated  use.  All  programed  combinations  of  initial 
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conditions  should  be  easy  to  change  off  line  so  they  can  be  kept  current 
with  changing  instructional  objectives  and  practices. 

Inputting  position  initialization  information  from  ATD  consoles 
should  involve  the  use  of  methods  that  aircrew  members  are  familiar 
with.  For  example,  several  cases  were  observed  where  an  instructor 
wished  to  initialize  the  ATD  at  a  particular  range  and  bearing  from  an 
established  reference  point,  in  this  case  the  end  of  a  runway.  The 
console,  however,  was  designed  to  accept  only  latitude  and  longitude 
inputs.  Thus,  it  was  necessary  for  the  instructor  to  use  supplemental 
charts  and  determine  latitude  and  longitude  corresponding  with  the 
desired  range  and  bearing  so  that  initialization  parameters  could  be 
updated.  Although  this  certainly  was  not  an  impossible  task,  it  was 
viewed  as  a  nuisance,  and  should  be  avoided  in  the  future. 

Users  of  several  ATDs  with  wide  angle  visual  systems  reconmended 
that  the  visual  system  be  automatically  blacked  out  during  ATD 
initialization  and  reinitialization.  Rapid  simulated  movement  of  the 
ATD  to  different  points  in  space  causes  rapid  movement  of  visual  system 
imagery.  This  rapid  movement  can  be  quite  distracting,  making  it 
necessary  in  some  cases  to  remind  pilots  in  the  cockpit  to  close  their 
eyes  or  look  away  from  the  visual  system  during  reinitialization. 

Not  every  facet  of  the  ATD  can  be  initialized,  at  least  with  current 
technology.  For  example,  it  often  is  necessary  to  manually  reposition 
throttles,  landing  gear  levers  and  other  cockpit  controls,  and  to 
re-trim  the  ATD.  Failing  to  do  so  can  result  in  very  irregular  flight 
performance  following  initialization.  A  number  of  ATD  users  have  found 
It  necessary  to  develop  checklists  of  manual  procedures  that  must  be 
performed  in  the  cockpit  during  reinitialization.  One  reasonable 
alternative  is  to  provide  instructors  with  electronic  display  of  these 
memory  aids  at  their  consoles. 

MANUAL  AIO  PROGRAMMABLE  MALFUNCTION  CONTROL 

Summary 

Emergency  procedures  training  Is  one  of  the  most  well  accepted  uses 
of  ATDs.  In  most  ATDs  surveyed,  simulated  malfunctions  and  emergency 
conditions  were  inserted  and  removed  manually  by  the 
Instructor/operator.  It  Is  technologically  possible  to  program  the 
occurrence  of  malfunctions  to  relieve  Instructors  of  the 
Insertion/removal  task.  Properly  Implemented,  the  concept  has  value. 
Programmed  malfunction  Insertion,  however,  has  not  been  well  received 
for  two  reasons.  The  first  Is  that  present  implementations  cause 
malfunctions  to  be  Inserted  at  predetermined  times  during  the  mission, 
but  mission  time  may  not  relate  to  mission  events.  Second,  some 
Instructors  express  concern  that  "word  will  get  out"  among  students  on 
what  malfunctions  will  occur  at  specific  points  during  ATD  training 
exercises. 
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A  new  approach  to  automatic  malfunction  insertion  is  being  tried  in 
an  experimental  prototype  instructor  console  for  a  Navy  F-14  OFT.  The 
approach  allows  the  instructor  to  select  malfunction  and  initiating 
condition  pairs  from  electronically  displayed  shopping  lists.  The 
initiating  conditions  are  mission  events,  such  as  weight  off  wheels  or 
the  passing  of  a  geographic  reference  point.  With  this  system,  the 
instructor  also  can  inhibit  an  automatic  malfunction  insertion,  and  can 
change  initiating  conditions  during  the  training  exercise.  This  system 
will  not  become  operational  until  late  1980,  so  its  acceptance  by 
instructors  remains  unknown  at  this  time.  However,  the  console  design 
is  intended  to  unburden  instructors  from  routine  tasks  so  they  can 
concentrate  on  more  important  facets  of  training. 

Feature  Description 

Procedures  trainers,  operational  flight  trainers,  part  task  trainers 
and  full  mission  trainers  almost  always  include  the  capability  to 
simulate  a  variety  of  malfunctions  so  that  training  in  recogni2ing  and 
responding  to  them  can  be  accomplished  in  a  safe,  controlled 

environment.  The  number  of  malfunctions  that  an  instructor  can  present 
to  the  student  varies  with  the  system  being  simulated.  A  typical  range 
is  from  60  to  several  hundred.  The  number  used  in  training  also  varies, 
but  typically  ranges  from  20  to  50,  and  includes  the  most  critical  and 
common  malfunctions  that  meaningfully  can  be  addressed  during  available 
training  time. 

The  most  common  method  of  Inserting  and  removing  simulated 

malfunctions  Is  the  manual  operation  of  controls  by  the  Instructor.  The 
types  of  controls  now  in  use  Include:  dedicated  pushbuttons; 
prograumable  pushbuttons;  alphanumeric  keyboards,  where  malfunctions  are 
entered  and  removed  based  on  a  number  code;  and  touch  panels,  where  the 
Instructor  simply  touches  a  display  of  options  to  select  the  desired 
malfunction.  Most  malfunction  insertion  and  removal  capabilities  also 
provide  the  Instructor  with  a  memory  aid  that  shows  which  malfunctions 
presently  are  active.  Cathode  ray  tube  (CRT)  systems  frequently  provide 

a  displayed  list  of  active  malfunctions.  Other  systems  Involving 

pushbuttons  Illuminate  legends  showing  which  malfunctions  are  active. 
Cases  were  observed  during  the  program  where  CRT  displays  were  added  to 
multi-crew  ATDs  to  provide  a  central  listing  of  active  malfunctions 
because  illuminated  malfunction  pushbuttons  were  widely  dispersed 
throughout  the  simulated  cockpit,  and  instructors  experienced  difficulty 
In  keeping  track  of  malfunctions  Inserted  by  other  instructors. 

Little  creative  thought  has  been  given  to  developing  meaningful 
methods  for  automatically  Inserting  malfunctions.  Time  Into  the  mission 
has  been  the  most  common  method  used  In  many  ATDs.  This  has  proved 
unworkable  because  mission  time  often  does  not  relate  to  mission  events. 
For  example,  one  ATD  was  observed  where  time  into  the  mission  did  not 
take  freeze  Into  account.  If  the  ATD  was  frozen  so  the  Instructor  could 
work  with  the  student,  the  malfunction  clock  kept  running.  As  a  result, 
a  new  malfunction  could  occur  when  the  system  was  unfrozen.  If  the 
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reason  for  freezing  the  ATD  was  to  work  with  the  student  on  an  existing 
malfunction,  having  a  new  malfunction  crop  up  when  the  ATD  was  unfrozen 
was  not  welcomed  by  many  instructors.  Alternatives  to  this  unworkable 
and  rigid  approach  are  presented  in  the  following  section  titled  Design 
Considerations. 

Related  Support  Features 

Automatic  malfunction  insertion  and  removal  requires  at  least  a 
rudimentary  automated  performance  measurement  capability.  Measurement 
is  required  for  malfunction  insertion  because  the  conditions  for 
initiating  the  malfunction  must  be  sensed  and  compared  against  insertion 
criteria.  Similarly,  if  the  malfunction  is  to  be  automatically  removed 
after  the  student  has  successfully  coped  with  it,  automated  measurement 
is  necessary  for  assessing  when  the  correct  procedures  have  been 
completed.  If  the  malfunction  is  to  remain  in  effect  (e.g.  engine  out), 
then  an  automated  performance  measurement  system  designed  to  evaluate 
student  performance  must  know  this  so  that  appropriate  standards  of 
flight  performance  can  be  used. 

Instructional  Values 


It  is  widely  accepted  that  ATDs  provide  a  safe  and  controlled 
learning  environment  for  training  adequate  response  to  malfunctions  and 
emergencies  that  may  be  encountered  in  flight.  This  is  the  basis  for 
training  of  malfunction  and  emergency  procedures  in  ATDs. 

One  assumed  value  of  automatic  malfunction  insertion  and  removal  is 
that  it  unburdens  the  instructor  from  routine  tasks,  thus  freeing  his 
attention  and  time  for  more  Important  instructional  activities,  such  as 
providing  guidance  and  feedback. 

A  second  value  involves  student  self  practice.  Training  practices 
vary  from  site  to  site.  However,  some  training  managements  encourage 
students  to  practice  tasks  in  ATDs  whenever  ATD  time  is  available,  even 
if  Instructors  cannot  be  present.  In  these  cases,  automatic  malfunction 
insertion  can  be  used  to  structure  training  sessions.  For  example,  the 
student  might  request  that  the  ATD  operator  program  the  ATD  to 
Incorporate  malfunctions  that  he  does  not  feel  confident  in  responding 
to,  or  that  are  scheduled  to  be  covered  in  the  next  training  session. 
The  element  of  "surprise"  or  unexpectedness  can  exist  at  least  with 
respect  to  timing.  However,  the  student  Is  provided  with  the  cues  that 
he  needs  to  learn  to  respond  to  and  still  Is  required  to  scan  for  the 
cues. 

Observed  Applications 

Manual  malfunction  Insertion  and  removal  was  observed  at  all  sites 
visited.  Automatic  malfunction  Insertion  and  removal  was  not  observed 
at  any  site  visited  during  the  program.  Automatic  capabilities  were 
observed  by  program  team  members  during  prior  programs  Involving  the 


Navy  F-14.  The  following  ATDs  also  will  include  automatic  malfunction 
insertion:  A-10;  F-16;  B-52 ;  and  C-130.  Boolean  logic  will  be  used  at 
least  in  the  C-130  (and  perhaps  the  others)  to  cause  malfunction 
insertion. 

Utility  Information 

This  section  addresses  four  Issues.  The  first  is  the  preparing  of 
students  for  productive  use  of  hands-on  training  in  ATDs.  The  second  is 
ease  of  use  of  malfunction  features  by  instructors  and  device  operators. 
The  third  is  the  need  for  memory  aids  on  malfunctions  that  already  are 
active.  The  fourth  is  alternatives  for  automatically  inserting  and 
removing  simulated  malfunctions. 

Current  training  practices  involving  simulated  malfunctions  assume 
that  only  hands-on  training  will  work.  This  raises  the  issue  of  whether 
students  really  are  prepared  to  benefit  from  expensive  hands-on  training 
in  ATDs.  Chapter  IV  of  STRES  volume  titled  Utilization  of  Aircrew 
Training  Devices,  addresses  cognitive  training  and  the  Impact  on  ATD 
training.  In  there  the  reader  will  find  commentary  on  the  need  for 
preparing  students  for  maximum  benefit  from  hands  on  training,  and  will 
find  alternative  training  methods  discussed.  For  example,  it  was 
observed  frequently  during  the  program  that  students  were  not  proficient 
In  Identifying  cues  associated  with  specific  malfunctions  at  the 
beginning  of  ATD  training.  Better  use  of  ATD  time  would  have  resulted 
if  prior  study  of  cues  and  their  meaning  had  been  emphasized. 

Instructor/operator  convenience  and  feature  ease  of  use  are 
important  Issues.  If  Instructors  do  not  care  to  use  an  available 
feature  because  it  is  troublesome  or  requires  special  attention,  then 
the  feature  typically  will  be  ignored.  Ease  of  use  of  manually 
Inserted/ removed  malfunctions  can  be  improved  In  several  ways.  First, 
the  malfunctions  must  be  clearly  identified.  ATDs  were  observed  where 
this  was  not  the  case.  This  particularly  was  true  with  programmable 
pushbuttons  used  for  Insert ion/ removal.  The  pushbuttons  were  not 
labeled  (since  their  function  was  changeable).  Instructors  either 
relied  on  device  operators  to  remember  what  button  operated  what 
malfunction,  wrote  notes  as  reminders,  or  simply  did  not  use  the 
programmable  controls. 

Memory  aids  are  required  to  remind  the  instructor  of  malfunctions 
already  Inserted.  This  was  found  to  be  particularly  true  In  multi-crew 
ATDs  where  several  Instructors  were  involved.  Keeping  track  of  what 
malfunctions  already  had  been  Inserted  required  a  memory  aid,  which  in 
most  cases  was  a  CRT  display. 

Automated  malfunction  Insert ion/ removal  was  not  used  for  reasons 
previously  cited.  However,  this  rejection  may  be  due  to  the  ways 
malfunctions  are  Inserted  automatically  In  present  ATDs.  Advanced 
designs  Incorporating  more  operationally  meaningful  Insert Ion/ removal 
logics  should  overcome  these  problems. 


65 


Related  Research 


Only  one  research  study  was  identified  that  addressed  manual  or 
automated  malfunctions.  The  report  (Semple,  Vreuls,  Cotton,  Durfee, 
Hooks  and  Butler,  1979)  acknowledged  the  present  shortcomings  of  manual 
and  automated  malfunction  insertion,  and  presented  alternative 
initiating  conditions.  The  initiating  conditions  they  identified  as 
potentially  valuable  centered  on  mission  events  (rather  than  mission 
time).  These  included  weight  off  wheels,  flap  retraction  or  extension, 
and  the  passage  of  geographic  points.  Their  design  also  allowed 
instructors  to  select  initiating  conditions,  and  override  initiating 
conditions  selected  at  the  beginning  of  the  training  session. 

Design  Considerations 

A  distinction  must  be  made  between  manually  controlled  malfunction 
insert  ion/ removal  and  automation  of  this  task.  From  the  student's 
standpoint,  training  to  deal  with  various  malfunctions  should  be  treated 
as  any  other  training  objective.  In  other  words,  training  management 
determines  what  malfunctions  are  worth  training  for,  and  training  in 
ATDs  should  focus  on  these  requirements.  Too  often,  however, 
malfunctions  are  inserted  to  "increase  the  student's  workload",  without 
any  specific  training  end-product  in  mind.  This  use  of  malfunctions  may 
be  beneficial,  but,  at  best,  it  is  unstructured  and  is  left  to 
Individual  discretion.  A  better  approach  would  be  to  treat  malfunction 
training  objectively.  For  example,  members  of  the  survey  team  noted 
that  malfunction  performance  required  during  standardization  and 
evaluation  tests  sometimes  was  not  practiced  during  training.  Thus, 
students  were  subjected  to  "final  exams"  where  they  were  required  to 
perform  procedures  not  incorporated  into  their  hands-on  training.  This 
reflects  poor  training  management.  A  well  designed  instructional 
support  system  would  keep  track  of  malfunctions  that  were  responded  to 
correctly,  those  requiring  additional  practice,  and  those  required  at 
the  end  of  training. 

Memory  aids  are  required.  ATD  designs  should  acknowledge  this  by 
providing  Instructor/operators  with  memory  aids  showing  what 
malfunctions  already  have  been  activated. 

AUTOMATED  CUING  AND  COACHING 

Summary 

Automated  cuing  and  coaching  are  not  common  in  present  ATDs.  Their 
assumed  Instructional  values  center  on  the  promptness  and  accuracy  of 
guidance  and  feedback  Information  provided  to  the  student,  and  the 
unburdening  of  instructors  through  automation.  Similarly,  automated 
cuing  and  coaching  systems  can  provide  feedback  and  guidance  to  students 
when  an  instructor  may  not  be  present.  The  systems  require  good 
automated  performance  measurement  because  automated  measurement  is  used 
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to  determine  what  messages  should  be  given  to  the  student  and  the  timing 
of  their  delivery.  Automated  cuing  and  coaching  systems  can  be 
disruptive  to  the  student  if  prompts  and  guidance  messages  occur  too 
frequently.  This  suggests  the  need  to  be  able  to  deactivate  the  system, 
and  further  suggests  the  desirability  of  decision  logics  designed  to 
keep  the  frequency  of  cuing  and  coaching  messages  within  acceptable 
bounds. 

Feature  Description 

A  cuing  message  is  one  that  prompts  the  student  by  identifying  a 
performance  deficiency.  An  example  is:  "check  your  altitude."  A 
coaching  message  directs  the  student's  behavior  through  verbal 
instruction.  An  example  is:  "return  to  flight  level  150". 

A  programmed  mission  scenario  is  required  so  that  desired 
performance  is  defined  clearly.  An  automated  performance  measurement 
(APM)  system  and  additional  decision  logics  also  are  required  for 
determining  message  content  and  timing.  An  APM  capability  is  needed  to 
sense  when  student  performance  is  less  than  what  is  required  for  the 
task  he  is  learning.  For  example,  departures  from  desired  speed, 
course,  heading  or  altitude  must  be  sensed.  (This  raises  the  issue  of 
performance  tolerance  bands  that  are  acceptable  for  different  stages  of 
training.  Further  research  is  needed  on  this  issue.)  When  differences 
are  found,  system  logics  are  needed  to  identify  the  appropriate  message 
content  to  be  transmitted  to  the  student.  Typically,  a  cuing  message 
would  be  transmitted  first.  Performance  monitoring  would  continue;  if 
the  performance  deficiency  was  not  corrected,  then  the  appropriate 
coaching  message  would  be  transmitted  to  the  student.  If  the 
performance  deficiency  continued,  either  the  ATD  could  be  frozen  or  the 
instructor  could  be  alerted  so  that  he  could  intervene. 

Three  technologies  are  available  for  creating  the  messages  to  be 
transmitted  to  the  student.  One  is  audio  tapes.  The  system  searches  a 
tape  file  for  the  appropriate  message,  and  then  plays  the  message  to  the 
student  via  the  ATD  intercom.  The  second  is  digitally  stored  speech. 
This  technology  provides  much  faster  message  delivery  than  is  possible 
using  tapes.  The  third  technology  is  computer  generated  speech. 
Computer  generated  speech  is  relatively  new  but  is  readily 
understandable.  Further  design  improvements  are  being  made  to  make 
computer  generated  speech  sound  less  machine-like.  This  digital 
technology  Is  well  suited  to  automated  cuing  and  coaching  messages 
because  the  messages  Involved  usually  are  brief,  and  computer  memory 
requirements  are  well  within  reason.  Also,  changes  In  message  content 
are  easily  accomplished. 

The  Issue  of  which  messages  should  be  built  Into  an  automated  cuing 
and  coaching  system  must  be  addressed  on  a  case  by  case  basis  depending 
on  the  training  being  provided  and  student  learning  problems  during 
training.  The  analysis  should  begin  by  identifying  typical  performance 
problems  that  occur  and  go  unattended  by  the  student.  Conmonalitles 


among  the  performance  problems  then  should  be  Identified  so  that  the 
smallest  possible  set  of  cuing  and  coaching  messages  can  be  identified. 
Draft  messages  then  should  be  developed  and  reviewed  for  clarity  and 
brevity.  Finally,  the  automated  cuing  and  coaching  system  should  be 
tried  out  in  an  operational  training  setting  before  its  design  is 
finalized. 

Related  Support  Features 

Relationships  of  automated  cuing  and  coaching  to  other  instructional 
support  features  are  summarized  below. 

Programmed  Mission  Scenarios.  A  programed  (i.e.  pre- specified) 
mission  scenario  Is  required  so  that  a  baseline  of  desired 
performance  exists.  The  programmed  scenario  can  be  as  simple  as  an 
ILS  profile,  or  can  Involve  an  entire  mission  plan  depending  on  the 
training  requirements  being  addressed. 

Automated  Performance  Measurement.  An  automated  performance 
measurement  capability  Ts  needed  to  sense  specific  performance 
deficiencies  so  that  correct  cuing  and  coaching  messages  can  be 
transmitted  to  the  student.  (See  Chapter  IV  of  this  report) 

Instructional  Values 


Assumed  instructional  values  for  automated  cuing  and  coaching  center 
on  the  promptness  and  accuracy  of  Information  provided  to  the  student, 
and  the  unburdening  of  Instructors  through  automation.  Similarly,  such 
systems  can  provide  feedback  and  guidance  to  students  when  an  instructor 
may  not  be  present. 

It  Is  not  uncommon  for  Instructors  or  console  operators  to  be 
distracted  temporarily  with  other  tasks  when  a  student's  performance 
begins  to  degrade.  Automated  cuing  and  coaching  can  get  around  this 
problem  and  make  sure  that  feedback  and  guidance  are  provided  to  the 
student  In  a  timely  manner.  Also,  at  a  number  of  training  facilities, 
students  are  encouraged  to  use  ATDs  on  their  own  to  practice  tasks. 
Frequently,  an  Instructor  Is  not  present.  Thus,  automated  information 
may  be  all  that  Is  available  to  the  student. 

Present  applications  of  automated  cuing  and  coaching  have  centered 
on  basic  (undergraduate)  flight  and  navigation  task  training.  In  the 
future.  It  may  be  possible  to  Incorporate  these  capabilities  into 
procedures  task  training.  Automated  measurement  of  student  performance 
of  normal  procedures  and  emergency  procedures  Is  advancing.  With  this 
capability.  It  should  be  possible  in  the  future  to  provide  automated 
cuing  and  coaching  assists  for  procedures  training. 
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Observed  Applications 

Only  one  application  of  automated  cuing  and  coaching  was  observed. 

The  application  involved  initial  entry  rotary  wing  pilot  training  on 
instrument  procedures.  An  automated  measurement  system  monitored  the 
following  parameters:  heading,  glidepath,  altitude,  airspeed,  bank 
angle,  pitch  angle,  trim,  vertical  velocity,  and  turn  rate.  Cuing 
messages  were  brief  and  consisted  of:  check  heading,  check  altitude, 
etc.  Coaching  messages  provided  guidance  to  the  students.  Typical 
coaching  messages  were:  adjust  heading  to  return  to  course;  and  adjust 
power  to  return  to  glide  path.  The  messages  were  stored  on  magnetic 
tape  and  were  presented  aurally  to  students.  The  instructor  had  the 
option  of  turning  the  system  off,  selecting  individual  parameters  to  be 
monitored  for  message  generation,  or  selecting  all  simultaneously.  A 
symbol  appeared  on  the  instructor's  pictorial  flight  profile  display 
each  time  the  system  transmitted  a  cuing  or  coaching  message.  This 
information  was  available  for  use  during  debriefing.  Use  of  the 
automated  cuing  and  coaching  system  was  at  the  instructor's  discretion. 

It  is  not  known  how  frequently  it  was  used.  A  number  of  student  pilots 
reported,  however,  that  they  felt  the  messages  were  valuable  during 
their  training. 

A  second  application  was  reported  by  Semple  (1974).  Undergraduate 
navigator  training  was  investigated  in  a  new  Navy  communication  and 
navigation  training  device.  The  device  incorporated  a  display  panel 
that  presented  cuing  messages  to  the  students.  A  tone  accompanied  each 
message.  This  cuing  system  was  found  to  be  so  disruptive  to  students 
that  it  was  deactivated.  It  must  be  noted  that  this  application 
involved  cognitive  skill  training  rather  than  flight  control  training. 

Also,  in  this  application  It  was  not  possible  to  selectively  eliminate 
some  of  the  cuing  messages.  Additionally,  in  this  case,  messages  were 
displayed  in  printed  form,  which  might  have  contributed  to  their  lack  of 
acceptance.  It  also  is  known  that  the  messages  occurred  too  often  .and 
that  the  simulation  was  not  frozen  when  performance  obviously  was 
deteriorating  and  Instructor  involvement  was  required.  In  other  words, 
automated  decision  logics  were  very  rudimentary  and  not  workable  in  this 
early  application. 

Utility  Information 

With  only  two  observed  applications.  It  is  difficult  to  make  solid 
statements  on  the  use  of  automated  cuing  and  coaching  for  the  broad 
spectrum  of  aircrew  training.  In  one  observed  application,  use  of  the 
feature  vaiMed  considerably  from  Instructor  to  instructor,  which  was  to 
be  expected  because  of  lack  of  Instructor  training  on  potential  values 
and  limits.  Some  Instructors  and  operators  indicated  that  the  feature 
was  used  "quite  often"  by  students  who  came  In  to  practice  on  their  own,  / 

but  they  could  not  quantify  the  amount  of  use.  The  observed  system  used 
audio  tape  recorded  messages.  Some  difficulty  was  experienced  Initially 
with  the  mechanical  reliability  of  the  tape  system.  This  problem 
eventually  was  overcome.  In  the  second  application,  poor  design  and 
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Implementation  made  clear  a  number  of  design  characteristics  to  avoid, 
but  precluded  determining  meaningful,  positive  guidelines. 

Related  Research 


No  systematic  research  was  found  that  addressed  the  instructional 
values  of  automated  cuing  or  coaching.  Similarly,  none  was  found 
dealing  with  performance  tolerance  bands  that  should  be  used  to 
trigger  messages  during  various  stages  of  skill  acquisition  or  retention 
training. 

Design  Considerations 

Again,  with  only  two  observed  applications,  it  is  difficult  to  make 
solid  design  recommendations.  The  following  guidelines  are  presented 
for  consideration.  They  are  based  largely  on  a  "common  sense"  approach 
and  must  be  subjected  to  test  and  evaluation. 

Computer  speech  generation  is  a  new  but  workable  technology.  It  is 
reasonable  to  use  this  technology  as  the  means  for  creating  cuing  and 
coaching  messages,  particularly  because  of  the  timeliness  of  the 
messages  that  can  be  generated. 

Automated  cuing  and  coaching  systems  can  inundate  students  and 
instructors  with  messages.  It  is  necessary,  therefore,  to  make  it 
possible  for  instructors/operators  to  selectively  activate  the  messages, 
rather  than  force  the  messages  on  them. 

It  also  is  probable  that  the  use  of  looser  performance  tolerances 
early  in  training  to  trigger  cuing  and  coaching  messages  is  desirable. 
This  could  serve  to  reduce  the  number  of  messages  transmitted  to  the 
student  during  early  training  when  his  abilities  to  perform  may  be 
significantly  less  than  what  is  expected  of  him  at  the  conclusion  of 
training.  However,  there  presently  are  no  guidelines  for  determining 
these  tolerances.  Further  research  is  needed. 

ATD -MOUNTED  AUDIO  VISUAL  MEDIA 

Summary 

Slide/tape  presentations  of  instructional  information  have  been  used 
for  many  years  by  commercial  airlines  as  a  complement  to  guidance 
provided  by  instructors  in  classrooms  and  cockpit  procedures  trainers. 
The  U.S.  military  now  is  beginning  to  use  these  media  to  provide 
systematic  Instruction  and  detailed  procedural  information  to  aircrew 
members  in  ATDs.  The  goals  are  to  provide  structured  procedural 
guidance,  complete  information,  and  to  do  so  in  a  context  where  pilot 
trainees  actually  can  perform  and  practice  the  procedural  steps  that  are 
required.  Additionally,  system  operational  information  can  be  presented 
and  reviewed.  Typically,  the  presenting  of  pictorial  information  and 
procedural  steps  has  been  through  projections  at  the  cockpit  windscreen. 
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Verbal  information  has  been  presented  via  a  tape  recording.  Such 
systems  have  been  well  received.  They  allow  structured  training  not 
only  when  an  instructor  is  present,  but  also  when  students  are  in  a  CPT 
or  OFT  without  an  instructor. 

Feature  Description 

Two  types  of  audio  visual  presentations  were  observed  during  the 
program.  Current  technology  used  slides  and  accompanying  audio  type 
presentations.  Older  technology  applications  were  16  millimeter  motion 
picture  presentations  with  sound.  With  either  medium,  the  content  of 
what  Is  presented  Is  of  primary  importance;  the  medium  is  simply  a  means 
of  conveying  the  message.  In  this  regard,  video  tape  and  video  disk 
technology  must  be  considered  as  media  alternatives. 

Related  Support  Features 

Audio  visual  media  are  not  directly  related  to  any  other 
instructional  support  features. 

Instructional  Values 


Two  instructional  values  commonly  are  assumed.  The  first  is 
consistency  and  completeness  of  the  information  presented.  Details  are 
not  overlooked,  and  standardization  of  instruction  is  promoted.  The 
second  is  an  efficiency  value.  With  cockpit-mounted  audio/visual 
training  media,  it  may  not  be  necessary  for  an  Instructor  to  be  present. 
For  example,  after  an  Initial  presentation  and  question/answer  period, 
students  may  benefit  from  Independent  study  and  practice  using 
audio/visual  presentations  as  a  source  of  guidance  and  as  a  baseline  for 
performance  assessment  and  feedback. 

Observed  Applications 

Two  applications  of  audio/visual  presentations  of  procedural 
training  Information  were  observed  during  the  present  program,  although 
the  authors  have  observed  others  during  prior  programs.  In  the  STRES 
program,  applications  were  observed  at  United  Airlines  Boeing  747 
transition  training  program,  and  the  Navy's  E-2  transition  training 
program. 

At  United,  a  B-747  CPT  serves  as  the  classroom  for  transitioning 
captains  and  copilots.  The  CPT  also  is  used  extensively  for 
transitioning  flight  engineers,  although  these  pilots  also  receive 
classroom  training.  United's  B-747  CPT  program  Is  centered  around  a 
windscreen  audio/visual  (slide/tape)  presentation  of  procedural,  system 
and  flight  operations  Information.  An  instructor  is  present  during 
United's  CPT  training.  The  audio/visuals  provide  supporting  pictorial 
and  text  Information  to  provide  primary  Instruction.  It  was  observed 
that  Instructors  frequently  Interrupted  the  media  presentations  to 
amplify  on  points  previously  found  not  to  be  clear,  review  information 
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with  the  transitioning  pilots,  and  to  present  and  explain  points  that 
were  new  (i.e.  the  media  presentations  were  outdated  or  Incomplete  in 
some  respects).  Pilot  responses  to  this  approach  to  training  were 
highly  favorable  for  several  reasons:  1)  instruction  was  highly 

personalized  since  the  typical  student  to  instructor  ratio  was  two  or 
three  to  one;  2)  abstract  information  (i.e.  system  design  and 
functioning)  could  be  clarified  through  direct  system  operation 
following  audio/visual  instruction;  3)  an  instructor  was  present  to 
amplify  on  and  clarify  information  as  well  as  answer  questions,  making 
the  instruction  highly  Individualized;  and  4)  the  pacing  of  instruction 
was  flexible  (within  limits)  to  pilot  needs.  Navy  instructors  voiced 
similar  viewpoints  with  respect  to  E-2  training.  Additionally,  the  Navy 
site  surveyed  also  allowed  and  encouraged  pilots  to  use  available  CPTs 
(and  other  ATDs)  on  a  time  available  basis,  even  though  Instructors  were 
not  present.  In  this  instance,  providing  structured  training  through 
cockpit-mounted  audio/visual  media  was  useful  to  the  students, 
especially  because  structure  to  the  training  was  provided  even  though  an 
instructor  might  not  be  present. 

Utility  Information 

The  value  (utility)  of  ATD-mounted  audio/visuals  has  been  described 
in  the  preceding  part  of  this  section. 

Related  Research 

No  research  was  found  that  dealt  with  the  effectiveness  or 
efficiency  of  ATD-mounted  audio/visual  media.  The  general  effectiveness 
of  such  media  commonly  Is  assumed,  however,  in  many  other  applications. 
Also,  efficiency  is  assumed  because  of  favorable  student  to  instructor 
ratios.  Prior  experience  shows,  however,  that  both  effectiveness  and 
efficiency  are  under  the  control  of  the  instructor.  If  he  forces  the 
pace  of  instruction  in  keeping  with  the  media  pace,  then  some  pilots 
will  be  "left  behind"  because  of  a  lack  of  understanding  of  what  was 
presented.  Obviously,  this  is  counterproductive  and  undesirable, 
although  It  happens.  On  the  other  hand.  If  the  pace  of  instruction  is 
too  slow,  lack  of  student  attention  can  result,  and  key  points  can  be 
missed.  This,  too,  can  lead  to  lack  of  training  effectiveness.  The 
issue  is  to  design  and  evaluate  each  training  (audio/visual)  module  so 
that  assurance  is  available  that  the  module  is,  indeed,  geared  to  the 
learner  audience.  Finally,  Instructor  training  becomes  Involved.  Most 
Instructors  have  the  technical  knowledge  required  to  amplify  upon  or 
explain  further  what  has  been  shown  through  audlo/visual  media. 
However,  some  are  prone  to  telling  "war  stories",  which  can  lead  to 
instructional  Inefficiencies  and  learner  boredom. 

Design  Considerations 

The  technical  correctness  of  any  audlo/visual  presentation  is  a 
product  of  the  thoroughness  and  technical  knowledge  of  those  who  prepare 
it.  The  attention  getting  value  of  the  presentations  remains  an  art 
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form  (although  budget  also  can  be  a  factor).  Without  question,  a  team 
of  experts  should  be  used  to  develop  the  technical  content  of  any 
audlo/vlsual  presentation.  A  team  Is  required  so  that  all  Important 
technical  content  Is  Identified.  A  qualified  Instructional 
technologist,  however,  should  direct  how  the  Information  Is  sequenced, 
presented  and  evaluated.  The  technologist  also  should  have  a 
demonstrated  capability  to  Implement  scripts,  visuals  and  feedback 
systems  since  all  of  these  factors  are  critical  to  Information  transfer, 
performance  (learning)  assessment  and  user  acceptance. 

One  problem  was  observed  with  cockpit  audlo/vlsual s  during  program 
site  visits,  and  the  problem  was  Independent  of  the  user  acceptance  of 
the  media.  It  was  observed  that  Instructors  often  had  to  Interrupt 
audio/visual  presentations  to  review  materials  and  answer  questions. 
With  sllde/tape  media,  backing  up  a  presentation  for  replay  Is 
cumbersome  because  It  Is  necessary  to  separately  back  up  the  slides  and 
the  tape,  and  then  re* synchronize  them.  Typically  what  was  done  was  to 
provide  the  Instructor  with  printed  text  of  the  script.  Although  this 
was  observed  to  have  worked  well,  It  leaves  the  problem  of 
re- synchronizing  the  backed-up  slides  with  the  audio  tape.  This  problem 
could  be  overcome  by  using  video  tape  or  video  disc  recordings,  although 
they  may  be  more  difficult  (costly)  to  update. 


CHAPTER  IV 

MONITOR  AND  EVALUATE  PERFORMANCE 


FUNCTION  DEFINITION 

This  chapter  addresses  two  instructional  functions  (monitor  and 
evaluate)  because  one  supports  the  other  in  Inseparable  ways. 
Evaluation  is  a  classic  decision  function.  It  is  determining  the 
relations  between  observed  performance  and  required  performance  through 
systematic  and  careful  comparisons.  Monitoring  is  the  observing  or 
gathering  of  relevant  performance  information  to  use  in  the  decision 
process.  Performance  is  the  observable  behavior  of  the  student  aircrew 
member  and/or  observable  system  responses  that  result  from  his  actions. 
Performance  observations  can  be  made  directly  by  the  instructor,  or 
indirectly  by  means  of  computer-based  automated  performance  measurement 
systems. 

RELATED  FUNCTIONS 

The  reader  is  directed  to  the  following  chapters  of  this  report  that 
address  functions  directly  related  to  monitoring  and  evaluating 
performance. 

Chapter  V,  Control  and  Individualize  Training 

Chapter  VIII,  Instructor/Operator  Console 
Oeslgn  and  Location 

Chapter  IX,  Instructor/Operator  Training 

The  reader  also  is  referred  to  Chapter  VIII  of  the  utilization  report, 
which  deals  with  assessing  ATD  training  effectiveness  and  Includes  a 
section  on  performance  measurement  for  effectiveness  assessment 
purposes. 

INSTRUCTIONAL  SUPPORT  FEATURES  ADDRESSED 

The  instructional  features  addressed  in  this  chapter  are: 

Automated  performance  measurement 
Automated  performance  alerts 
Annunciators  and  repeater  Instruments 
Closed  circuit  television 
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AUTOMATED  PERFORMANCE  MEASUREMENT 
Summary 

Automated  performance  measurement  (APM)  for  training  is  the 
computer-based  application  of  technology  to  monitoring,  recording, 
processing  and  displaying  objective,  quantitative  information  that 
describes  student  performance  and  assists  in  diagnosing  student  learning 
problems.  APM  systems  have  been  fairly  common  In  research  simulators 
for  over  ten  years.  These  APM  systems  typically  are  not  suited  for 
operational  training,  however,  because  they  have  been  tailored  for 
research  use  and  frequently  produce  performance  data  that  require 
subsequent  statistical  processing.  An  APM  system  designed  for  use  in 
training  must  perform  all  statistical  and  other  processing  of 
performance  data  in  real  or  near-real  time  so  that  students  and 
Instructors  are  provided  with  useful,  concise  and  timely  performance 
feedback  information. 

Practically  all  automated  measurement  capabilities  in  existing  ATDs 
or  ATDs  soon  to  be  delivered  are  best  described  as  performance 
monitoring  and  data  collection  systems.  These  capabilities  allow 
instructors  to  select  tolerance  bands  (e.g.  +/-  100  feet)  around  various 
performance  parameters  (e.g.  altitude).  The  performance  monitoring 
capability  then  monitors  for  cases  that  exceed  the  tolerance  bands 
values,  and  records  out  of  tolerance  conditions  for  subsequent  display 
at  the  Instructor's  console  or  for  hard  copy  printouts.  Such 
rudimentary  performance  monitoring  capabilities  have  been  used 
effectively  to  drive  automated  performance  alerting  systems  and 
automated  cuing  and  coaching  systems.  However,  they  have  found  little 
acceptance  for  performance  evaluation  and  learning  problem  diagnosis 
during  training.  In  other  words,  such  systems  are  not  used  by 
Instructors.  One  reason  is  that  the  volume  of  data  produced  by  such 
systems  often  Is  overwhelming  and  very  difficult  to  integrate  and 
Interpret.  A  second  possible  reason  is  that  Instructor  training  has  not 
addressed  the  use  of  such  data. 

Recent  APM  research  suggests  yet  another  reason  why  outputs  from 
parameter  monitoring  capabilities  are  frequently  not  used.  It  now  is  an 
accepted  fact  in  the  APM  research  and  development  community  that  aircrew 
performance  measurement  is  a  multi-variate  issue.  This  means  that 
performance  is  best  described  by  analysis  of  how  many  different 
performance  dimensions  simultaneously  change  with  respect  to  each  other, 
rather  than  how  they  change  individually.  Determining  these 
relationships  and  developing  rules  for  weighting  and  combining  many 
Individual  parameters  into  a  new,  statistically  defined  performance 
score  requires  the  use  of  sophisticated  statistical  procedures.  It  is 
not  surprising,  therefore,  that  providing  instructors  with  what  is 
basically  raw  performance  data  has  not  proved  helpful  for  evaluating  and 
diagnosing  the  student's  progress  in  training. 


APM  for  training  still  requires  experimental  development  work  for 
each  application.  Discussions  with  one  of  the  respected  researchers  In 
the  training  APH  field  revealed  the  following.  He  would  be  quite 
unwilling  to  prepare  an  APH  system  specification  unless  the 
specification  clearly  required  a  measurement  development  and  refinement 
process.  In  this  process,  actual  student  performance  data  would  be 
collected,  analyzed  and  used  to  refine  the  Initial  APH  system  so  that  It 
would  produce  a  minimal  set  of  valid  and  usable  measures  of  performance. 
Automated  measurement  of  human  performance  Is  complex,  the  measures  are 
Interactive,  and  the  technology  has  not  yet  been  thoroughly  developed. 

Feature  Description 

Automated  performance  measurement  (APH)  for  training  Is  the 
computer-based  application  of  technology  to  monitoring,  recording, 
processing  and  displaying  objective,  quantitative  Information  that 
describes  student  performance  and  assists  In  diagnosing  student  learning 
problems.  Modern  APM  systems  operate  In  real  time  or  near-real  time  so 
that  student  performance  Information  Is  quickly  available  for  feedback 
to  Instructors  and  students. 

It  Is  necessary  to  distinguish  automated  performance  measurement 
systems  from  automatic  data  monitoring  and  recording  systems.  Automatic 
data  monitoring  and  recording  systems  sample  the  values  of  parameters 
(e.g.  altitude  or  the  position  of  a  particular  control)  and  also  may 
monitor  for  cases  that  exceed  tolerance  band  values  that  have  been  set 
for  the  parameters.  Automatic  monitoring  and  recording  systems  then 
simply  store  and/or  display  this  Information  either  for  subsequent 
statistical  analysis  or  for  visual  examination  by  Instructors  and 
students.  The  few  "automated  measurement  systems"  that  are  found  In 
existing  ATDs  actually  are  automated  monitoring  and  recording  systems. 
They  collect  and  display  rather  rudimentary  data  forms.  No  data 
processing  (l.e.  transforming,  weighting,  combining  and  scaling)  Is 
provided.  All  of  these  functions  are  left  to  the  Instructor  who  must 
perform  them  subjectively.  Four  such  "automated  measurement  systems" 
were  observed  during  program  site  visits.  One  was  a  part  of  an  F-15 
OFT;  a  second  was  part  of  a  U.S.  Coast  Guard  helicopter  OFT;  the  third 
was  part  of  an  Army  rotary  wing  OFT;  and  the  fourth  was  Incorporated 
Into  a  Navy  F-14  OFT.  None  of  these  systems  was  used.  (See  the 
Utility  Information  part  of  this  section.) 

True  APM  systems  for  aircrew  training  applications  represent  an 
emerging  technology.  Recent  developments  In  computer  technology  have 
accelerated  APM  research  and  development,  but  automated  human 
performance  measurement  technology  Issues  still  require  considerable 
clarification  (Vreuls  and  Wooldridge,  1977;  and  Waag  and  Knoop,  1977). 

APM  systems  for  training  must  be  designed  to  support  specific 
training  objectives.  Thus,  the  specific  design  of  an  automated 
performance  measurement  system  will  depend  on  Its  training  application. 
General  characteristics  can  be  described,  however,  and  one  such  set  of 
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characteristics  Is  how  individual  measures  need  to  be  defined.  Based  on 
recent  APM  training  research,  it  appears  that  each  measure  must  be 
defined  in  terms  of  the  five  determinants  described  below  (Vreuls  and 
Wooldridge,  1977). 

Measure  Segment.  This  is  any  portion  of  a  maneuver  or  mission 
Tor  which  student  performance  or  system  performance  is 
relatively  constant  or  follows  lawful  relationships  from 
beginning  to  end,  and  for  which  the  beginning  and  end  can  be 
defined  unambiguously.  Measure  segments  can  overlap  or  can  be 
one-time  events.  The  measure  segment  must  be  defined  by 
explicit,  unambiguous  measurement  start/stop  logic.  This  has 
proved  to  require  sophisticated,  innovative  logic  designs. 

State  Variable.  A  state  variable  is  any  quantifiable  index  of: 
1)  vehicle  states  in  any  reference  plane  (e.g.  bank  angle);  2) 
student  physiological  states  (e.g.  heart  rate);  or  3)  control 
device  states  (e.g.  control  stick  or  switch  positions). 

Sampling  Rate.  This  is  the  temporal  frequency  at  which  a  state 
variable  (parameter)  is  recorded  or  examined  by  an  automated 
measurement  system. 

Desired  Value.  In  many  cases,  a  state  variable  by  itself  is  of 
little  measurement  use.  Frequently  it  is  necessary  to  compare 
the  state  variable  with  a  desired  value  to  obtain  an  error  or 
deviation  score.  Desired  values  can  be  determined  analytically 
or  experimentally. 

Transformation.  A  transformation  is  defined  as  a  mathematical 
treatment  of  the  error/deviation  score.  Transformations  can  be 
as  simple  as  just  the  state  variable's  value  or  its  absolute 
value,  or  may  Include  computations  of  out  of  tolerance 
conditions,  measures  of  central  tendency,  variability, 
frequency  content  or  departures  from  norms.  Common  transforms 
are  shown  In  Table  7. 

Measurement  transforms  also  do  the  job  of  summarizing  vast  numbers 
of  measurement  samples  Into  something  that  is  instructionally 
manageable.  For  example,  if  the  state  variable  being  measured  was  angle 
of  attack,  and  It  was  sampled  twice  per  second  during  a  three  minute 
Interval  (e.g.  during  final  approach  to  landing),  360  measure  samples 
would  result.  A  measure  transform  that  computed  the  variability  of 
differences  between  desired  angle  of  attack  versus  observed  angle  of 
attack,  averaged  across  the  entire  measurement  segnent,  would  result  In 
only  one  number.  Such  a  transformation  could  be  obtained  by  computing 
the  standard  deviation  of  angle  of  attack  deviation  values.  Thus, 
transformations  make  automated  performance  measurement  data  manageable, 
among  other  things.  (The  example  also  points  out  the  need  for 
developmental  research  of  APM  capabilities,  because  it  would  be 
analytically  impossible  to  establish  an  acceptable  value  for  such  a 
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Table  7.  Common  Measure  Transforms 


Time  History  Measures 
Time  on  target 
Time  out  of  tolerance 
Maximum  value  out  of  tolerance 
Response  time,  rise  time,  overshoot 
Frequency  domain  approximations 

Count  of  tolerance  band  crossings 
Zero  or  average  value  crossings 
Derivative  sign  reversals 
Damping  ratio 

Amplitude-Distribution  Measures 
Mean,  median,  mode 
Standard  deviation,  variance,  range 
Minimum/maximum  value 
Root-mean-squared  error 
Absolute  average  error 

Frequency  Domain  Measures 

Autocorrelation  funct 1 on 
Power  spectral  density  function 
Bandwidth 
Peak  power 

Low/high  frequency  power 
Bode  plots,  fourier  coefficients 
Amplitude  ratio 
Phase  shift 

Transfer  function  model  parameters 
Quasi-1 Inear  describing  function 
Cross-over  model 
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measure,  and  it  still  is  necessary  to  establish  experimentally  whether 
such  "unusual"  measures  in  fact  are  good  performance  descriptors.) 

APM  research  has  shown  that  some  measures  contribute  more  to  the 
total  description  of  student  performance  than  do  others.  The  research 
also  has  shown  that,  individually,  various  measures  may  not  be  useful 
for  discriminating  between  "good"  and  "poor"  performance,  but  when  they 
are  properly  weighted  and  combined,  the  resulting  measure  set, 
collectively,  is  quite  useful  for  discriminating  between  performance 
differences  (Waag,  et.  al.,  1975).  The  extent  to  which  various 
automated  performance  measures  must  be  weighted  and/or  combined  to 
become  useful  for  training  purposes  remains  a  research  issue.  The  need 
to  do  so  has  been  demonstrated,  however,  for  basic  instrument  flight 
maneuver  training  (Vreuls,  et  al.,  1976)  and  for  one  versus  one  air 
combat  maneuvering  training  (Kelly,  Wooldridge,  Hennessy,  Vreuls, 
Barnebey,  Cotton,  1979).  It  is  fairly  certain,  therefore,  that  at 
least  some  future  ATD  APM  systems  will  require  sophisticated,  near-real 
time  statistical  processing  of  multiple  as  well  as  individual  measures 
in  order  to  provide  instructors  and  students  with  valid,  usable 
performance  scores. 

Automated  measurement  of  procedural  performance,  such  as  the 
performance  of  checklist  items,  also  is  an  emerging  technology,  and  may 
not  be  seen  in  operational  applications  for  at  least  five  years. 
Automated  measurement  of  performance  of  procedural  sequences  requires 
very  complicated  computer  logics  which  only  now  are  being  developed  for 
training  applications  (Semple,  et  al.,  1979).  A  number  of  newly 
acquired  ATDs  will  incorporate  procedure  monitoring  capabilities, 
however  (e.g.  F-16,  A-10,  F-5,  B-52  and  C-130).  These  systems  will 
display  (on  either  cathode  ray  tubes  or  hard  copy  printouts)  the  actual 
sequences  in  which  procedures  are  peformed.  It  will  be  the  instructor's 
responsibility  to  determine  whether  or  not  the  procedural  sequences  and 
timing  were  acceptable. 

Related  Support  Features 

Some  form  of  automated  performance  measurement  is  essential  for  the 
use  of  the  instructional  support  features  described  below.  The  level  of 
APM  sophistication  required  for  each  feature  varies  significantly,  as 
noted. 

Automated  Adaptive  Training  (AAT).  AAT  is  training  in  which  the 
problem,  the  cues  or  the  task  are  varied  automatically  as  a 
function  of  how  well  the  student  performs.  Automated  performance 
measurement  Is  necessary  for  AAT  systems  to  determine  how  well  the 
student  Is  performing  and  to  identify  specific  performance  strengths 
and  deficiencies  so  that  subsequent  training  can  be  tailored 
automatically  to  the  student's  needs.  It  Is  believed  that  AAT 
systems  for  aircrew  training  will  require  highly  sophisticated 
automated  measurement  capabilities.  (See  Appendix  B  of  this 
report . ) 
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Computer  Managed  Instruction  (CMI).  CMI  systems  that  are  capable 
of*  tracking  the  student's  achievement  of  training  objectives  require 
automated  measurement  system  inputs  if  the  CMI  systems  are  to  be 
automated.  The  level  of  measurement  system  input  to  the  CMI  system 
can  be  quite  rudimentary  and  can  indicate  simply  that  a  particular 
training  objective  either  has  or  has  not  been  achieved  (i.e.  a 
yes-no  input).  The  system  measurement  requirements  for  making  the 
yes-no  decision,  however,  may  be  fairly  extensive,  depending  on  the 
training  objectives  that  are  involved.  (See  Chapter  VII  of  this 
report  for  additional  discussion  of  CMI  systems.) 

Automated  Performance  Alerts.  Automated  performance  alerts  are 
signals  provided  to  instructors  and/or  the  students  indicating  that 
some  facet  of  the  student's  performance  has  exceeded  accceptable 
limits.  Automated  measurement  is  required  if  such  alerts  are  to 
occur  automatically.  The  sophistication  of  the  measurement  that  is 
required  depends  on  nature  of  the  alert  and  what  is  required  to 
trigger  it.  An  automated  alert  that  indicates  simply  a  departure 
from  an  assigned  altitude  poses  minor  requirements  on  an  automated 
performance  measurement  system.  An  automated  alert  indicating 
improper  primary  flight  control  technique,  on  the  other  hand, 
requires'  considerably  more  complex  measurement.  (See  the  next 
section  of  this  chapter  for  further  information  on  automated 
alerts.) 

Automated  Cuing  and  Coaching.  Like  automated  alerts,  these 
instructional  support  capabilities  require  automated  measurement  as 
the  triggering  mechanism.  And,  as  with  automated  alerts,  the 
complexity  of  the  measurement  capability  required  to  operate 
automated  cuing  and  coaching  capabilities  depends  entirely  on  the 
nature  and  complexity  of  the  task  being  trained.  (See  Chapter  III 
of  this  report  for  additional  information  on  automated  cuing  and 
coaching.) 


Instructional  Values 


To  a  great  extent,  the  instructional  values  of  automated  performance 
measurement  are  the  same  as  the  values  of  measurement  In  general.  The 
general  values  of  measurement  are  to  provide  decision  making  data  to 
policy  makers,  technology  managers,  scientists,  training  managers. 
Instructors,  and  operational  commanders.  The  right  kinds  of  human 
performance  data  are  useful  in  guiding  decisions  about  doctrine, 
strategy,  tactics,  personnel  selection,  training,  training  device  and 
curriculum  design,  skill  maintenance,  proficiency  advancement,  and 
operational  readiness. 

Values  offered  by  automated  performance  measurement  for  aircrew 
training  Include:  precision;  objectivity;  standardization;  Improved 
record  keeping  (e.g.  for  trend  information);  and  the  capability  to 
measure  considerably  more  facets  of  human  performance  simultaneously. 
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The  need  for  improved  aircrew  performance  measurement  capabilities, 
including  automated  performance  measurement,  has  been  recognized  for 
some  time  (e.g.  Smode  and  Meyer,  1966;  and  Department  of  Defense,  1968). 
However,  it  only  has  been  recently  that  the  operational  environment  has 
come  to  recognize  the  need  for  improved  measurement  (Waag  and  Knoop, 
1977).  The  needs  are  even  more  pressing  now  because  of  severe  budgetary 
constraints  and  fuel  realities.  These  factors  imply  needs  for 
improvements  in  aircrew  training  efficiency  and  effectiveness,  skill 
maintenance  and  operational  readiness..  Improved  efficiencies  require 
improved  measurement.  Responsiveness  to  these  needs  is  reflected  in 
continuing  automated  performance  measurement  development  programs  in  the 
Air  Force,  Navy  and  Army. 

Observed  Applications 

The  automated  measurement  systems  observed  during  this  program  are 
best  described  as  performance  monitoring  and  recording  capabilities 
rather  than  true  automated  performance  measurement  systems.  Performance 
monitoring  and  researching  capabilities  monitor  whether  tolerance  bands 
(e.g.  +/-  5  degrees)  around  various  performance  parameters  (e.g. 
heading)  are  exceeded.  Cases  that  exceed  tolerance  band  values  are 
recorded  for  subsequent  display  at  the  instructor's  console  or  for  hard 
copy  printout. 

Four  such  performance  monitoring  and  recording  capabilities  were 
observed;  one  was  incorporated  into  an  Amy  rotary  wing  OFT;  the  second 
was  part  of  a  U.S.  Coast  Guar$  helicopter  OFT;  the  third  was  part  of  an 
Air  Force  F-15  OFT;  and  the  fourth  was  part  of  a  Navy  F-14  OFT.  The 
first  device  was  used  primarily  for  basic  pilot  training.  The  second, 
third  and  fourth  devices  were  used  for  transition  and  continuation 
training. 

Utility  Information 

None  of  the  performance  monitoring  and  recording  systems  surveyed 
during  the  program  was  used  during  routine  training  operations.  Many  of 
the  monitoring/ recording  capabilities  shared  the  following  shortcomings: 

Set>up  of  the  capability  was  time  consuming  and  required  frequent 
changes.  For  example,  monitoring  how  well  a  pilot  maintained 
altitude  during  cruise  required  one  tolerance  band  setting;  checking 
to  see  whether  a  minimum  altitude  was  exceeded  during  an  Instrument 
approach  required  making  a  tolerance  band  change.-  Such  manual 
changes  to  the  system  are  a  nuisance  to  Instructors.  Therefore, 
they  do  not  make  them.  If  they  do  not  make  the  changes,  the 
performance  monitoring  capability  produces  Irrelevant,  meaningless 
data.  This  Is  one  reason  such  systems  are  not  used. 

The  data  that  could  be  produced  were  overwhelming  In  volune  and  were 
difficult  to  Interpret.  Lack  of  appropriate  Instructor  training 
may  have  contributed  to  the  problem.  However,  recent  automated 
performance  measurement  reasearch  suggests  another  reason  why  such 
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outputs  are  not  useful.  It  now  is  an  accepted  fact  in  the  APM 
research  and  development  community  that  performance  measurement  is  a 
multi -variate  issue.  It  has  been  found  that  aircrew  performance  is 
best  described  in  terms  of  how  a  number  of  different  performance 
measures  change  individually.  What  this  says  is  that  aircrew 
performance  is  as  complex  as  instructors  always  have  said  it  is,  and 
simplistic  (one  measure  at  a  time)  approaches  to  measurement  simply 
cannot  be  used  to  get  a  practical  handle  on  the  complexity  of 
aircrew  member  and  total  weapon  system  performance. 

Related  Research 


Practically  all  APM  work  has  been  done  in  the  context  of  automated 
meausrement  research  and  development.  Exceptions  include  the 
development  of  APM  capabilities  for  aircraft  control -display  research 
(e.g.  Monroe,  Vreuls  and  Semple,  1968)  and  aircrew  training  research 
(e.g.  Waag  and  Knoop,  1977;  and  Kelly,  M.  et  al.,  1979).  Even  these 
applications  have  focused  on  research  tasks  rather  than  operational 
performance  assessment  applications.  Exceptions  exist,  of  course. 
McDonald,  Smith,  Evans,  Baer  and  Nelson  (1979)  have  reported  the 
development  and  test  of  limited  APM  capabilities  for  the  F-15  OFT. 
Semple  et  al .  (1979)  report  the  design  of  a  limited,  experimental 
prototype  automated  performance  measurement  and  scoring  capability  for  a 
Navy  F-14  OFT.  This  prototype  system  will  be  used,  in  part,  for 
performance  measurement  development  work  in  an  operational  (transition 
and  continuation)  training  setting.  Povernmire,  Russell  and  Schmidt 
(1977)  also  have  reported  a  study  in  which  rudimentary  types  of 
automated  measures  were  explored  to  determine  U.S.  Coast  Guard 
helicopter  pilot  proficiency.  All  of  the  studies  cited  above  have  dealt 
with  individual  aircrew  member  (usually  pilot)  performance.  Only  one 
program  is  known  to  be  addressing  APM  for  the  much  more  difficult  task 
of  assessing  crew  (versus  individual )  performance .  This  program 
involves  measurement  of  Air  Force  C-5  crew  performance. 

The  studies  cited  above  (and  other,  related  studies)  have  focused  on 
the  following  issues,  all  of  which  are  relevant  to  developing  a  workable 
APM  technology  base: 

How  should  measures  be  defined  in  terms  of  computer  software  and 
associated  logics? 

Which  automated  measures  validly  describe  aircrew  performance?  For 
example,  which  measures  actually  discriminate  between  obviously 
"good"  and  "poor"  performance? 

What  are  the  best  ways  to  weight  and  combine  individual  measures  to 
better  describe  and  predict  total  pilot  performance? 
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Which  measures  and  combinations  of  measures  are  most  useful  to 
instructors  for  diagnosing  student  learning  and  performance 
problems,  in  order  to  speed  up  learning? 

Which  measures  and  combinations  of  measures  are  useful  in  automated 
diagnosis  of  student  learning  problems,  quantifying  proficiency 
levels,  and  prescribing  subsequent  training  (i.e.  automated  adaptive 
training  applications)? 

Which  automated  measures  found  to  be  useful  in  training  research  and 
development  will  be  useful  in  day-to-day  training  operations? 

How  can  technology  developed  for  automated  measurement  of  flight 
performance  be  used  to  measure  the  performance  of  sequences  of 
procedural  events  (e.g.  checklists)? 

Which  automated  measures  (and  weighted  combinations  of  measures) 
correlate  highly  with  skilled  instructor  evaluations  (i.e.  which 
measures  will  be  perceived  as  valid  by  instructors)? 

How  should  APM  data  be  presented/displayed  to  instructors  and 
students? 

The  issues  identified  above  are  numerous  and  broad  ranging.  In 
addition,  the  number  of  aircrew  tasks  involved  is  great;  and  the  range 
of  pilot  skill  levels  to  be  assessed  covers  the  spectrum  from 
undergraduate  through  operationally  ready.  With  these  points  in  mind, 
there  should  be  little  surprise  that  APM  research,  which  has  been 
undertaken  seriously  only  within  the  past  decade,  has  not  led  to  the 
resolution  of  many  aircrew  measurement  needs.  The  following  paragraphs 
highlight  research  findings  that  are  representative  of  the  present  state 
of  the  art.  The  reader  is  reminded  that  this  is  an  area  requiring 
considerable  additional  research  and  development. 

Methods  for  defining  automated  measures  of  flight  performance  are 
fairly  well  accepted,  and  are  made  up  of  the  five  measure  determinants 
previously  described  In  the  section  titled:  Feature  Description. 

Defining  unambiguously  computer  logics  for  automatically  starting  and 
stopping  measurement  remains  an  intricate  and  sometimes  illusive  task. 
McDonald  et.  al.  (1979),  for  example,  reported  that  relatively  simple 
automated  measures  of  pilot  performance  during  a  standard  Instrument 
departure  correlated  highly  (r  ■  .75)  with  Instructor  ratings  of  pilot 
performance.  However,  automated  measurements  of  performance  during 
TACAN  approachs  or  ILS  final  approaches  were  either  uncorrelated  or 
negatively  correlated  with  Instructor  ratings.  McDonald  et  al. 
attribute  the  latter  findings  to  the  fact  that  their  automated 
measurement  system  started  measuring  performance  at  Inappropriate  times 
because  pilots  do  not  fly  the  text  book  approaches  their  measurement 
system  was  patterned  after. 
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It  is  relatively  easy  to  measure  many  facets  of  pilot  performance 
simultaneously  with  computer  based  measurement  systems.  This  can  result 
in  a  "measure  overload."  The  issue  then  becomes  one  of  identifying 
which  of  the  candidate  (experimental)  measures  best  describe 
performance,  and  which  do  not.  Multi-variate  statistical  procedures 
have  proved  to  be  valuable  tools  in  this  process.  It  has  been  shown, 
for  example,  that  initial  candidate  measure  sets  containing  30  or  so 
measures  can  be  reduced  to  about  10  measures  with  no  apparent  loss  in 
measurement  precision  (e.g.  Kelly,  M.  et  al.,  1979).  Appendix  B  deals 
with  this  issue  in  greater  detail.  Occasionally,  the  unexpected  also  is 
demonstrated.  For  example,  Vreuls,  Obermeyer,  Goldstein  and  Norman 
(1975)  showed  that  approximately  40%  o*  student  performance  variance  was 
accounted  for  by  measures  of  primary  flight  control  (e.g.  control  stick) 
inputs  in  a  task  involving  basic  instrument  flight  maneuvering. 

APM  research  using  multi-variate  statistical  procedures  also  has 
shown  that  not  all  measures  contribute  the  same  (i.e.  have  the  same 
weight)  to  describing  pilot  performance.  This  is  not  an  unexpected 
finding,  since  instructors  often  note  that  one  or  more  facets  of 
performance  are  more  important  than  others  in  accomplishing  various 
flying  tasks.  In  automated  performance  measurement  systems,  however,  it 
is  necessary  to  know  fairly  precisely  just  how  much  weight  should  be 
assigned  to  individual  measures.  Again,  multi-variate  statistical 
techniques  can  be  of  considerable  value  in  determining  appropriate 
weights  based  on  analysis  of  representative  samples  of  actual 
performance  data  (See  Appendix  B  of  this  report).  In  lieu  of  having 
empirically  determined  performance  weights  to  use,  some  researchers  have 
assigned  weights  to  different  performance  measures  based  on  expert 
opinion  (e.g.  McDonald,  et  al.,  1979).  The  problem  is  that  such  weights 
may  not  be  valid,  and  may  distort  performance  appraisals  (McCauley  and 
Semple,  1980).  Considerable  further  research  is  needed  to  determine  the 
relative  importance  of  specific  measures  for  performance  description  and 
learning  problem  diagnosis  in  the  training  of  the  many  tasks  required  in 
military  missions. 

Very  little  is  known  about  designing  APM  systems  specifically  for 
learning  problem  diagnosis  so  that  automated  adaptive  training  system 
logics  can  function  most  efficiently.  Wooldridge,  Vreuls  and  Norman 
(1977)  reported  a  study  that  compared  several  different  automated 
adaptive  logics  (training  strategies).  The  measurement  system  they  used 
obviously  was  quite  complex,  but  was  not  fully  reported  by  them.  The 
tasks  they  studied  were  basic  instrument  pilot  maneuvers,  and  relatively 
inexperienced  pilots  were  used  as  subjects.  Although  their  automated 
adaptive  training  logics  seemed  to  operate  as  designed  (which  implies 
that  the  automated  measurement  system  also  was  adequate),  further 
research  also  is  required  in  this  area  because  of  the  limited  nature  of 
the  tasks  used  in  research  to  date. 


Waag  and  Knoop  (1977)  report  that  automated  measurement  methods 
exist  for  measuring  performance  on  a  variety  of  Air  Force  undergraduate 
pilot  training  tasks.  The  measurement  system  is  implemented  in  the 
Advanced  Simulator  for  Pilot  Training  (ASPT)  at  Williams  AFB.  Whether 
these  research  measurement  methods  can  be  applied  and  used  practically 
in  undergraduate  pilot  training  operations  has  not  yet  been 
demonstrated,  although  they  may  be.  Similarly,  Kelly,  M.  et  al.  (1979) 
have  reported  the  development  and  test  of  an  initial  APM  capability  for 
measuring  facets  of  one  versus  one  air  combat  maneuvering  performance  in 
the  Air  Force  SAAC  simulator  at  Luke  AFB.  The  Vought  Corporation  also 
has  been  tasked  by  the  Air  Force  to  develop  the  "Good  Stick  Index"  for 
test  and  evaluation  in  air  combat  training  simulators.  Whether  any  of 
these  new  but  complex  measurement  developments  prove  workable  in 
day-to-day  training  applications  remains  to  be  established.  At  this 
point,  further  developmental  research  and  validation  is  needed  to  answer 
the  question. 

The  issue  of  applying  automated  measurement  techniques  to  measuring 
the  performance  of  sequences  of  procedural  events  (e.g.  pre-start 
checklist  procedural  sequences)  is  just  now  receiving  attention.  In 
this  application,  an  experimental  prototype  instructional  support  system 
for  a  Navy  F-14  OFT  will  incorporate  limited  monitoring,  measurement  and 
scoring  of  procedural  performance  by  a  single  crewmember.  Preliminary 
indications  are  that  considerable  and  detailed  task  analysis  must 
precede  the  development  of  computer  measurement  logics.  This  is  because 
correct  sequences  must  be  determined  in  detail,  and  likely  alternative 
sequences  (both  acceptable  and  unacceptable)  also  must  be  defined  so 
that  computer  measurement  and  scoring  logics  act  fairly  and  do  not 
penalize  pilots  for  using  occasional  (but  acceptable)  departures  from 
normal,  textbook  procedural  sequences.  Related  to  this  are  the  complex 
computer  logics  required  to  monitor  and  track  procedural  sequence 
chains,  decide  on  their  appropriateness,  and  measure  and  score 
appropriately.  Initial  research  data  should  become  available  by  early 
1981. 

The  preceding  paragraphs  of  this  section  certainly  have  not 
exhausted  the  research  on  this  relatively  new  and  highly  complex  topic. 
Rather,  the  intent  has  been  to  highlight  the  kinds  of  research  that  have 
been  accomplished  and  that  are  under  way.  The  fact  that  considerable 
further  work  is  required  is  recognized  by  all  three  services.  Because 
of  the  complexity  of  the  automated  measurement  research  and  development 
task,  the  Air  Force  Office  of  Scientific  Research,  with  tri-service 
coordination  and  support,  is  sponsoring  a  program  to  develop  APM 
standards.  One  primary  goal  is  to  explore  ways  of  making  individual  APM 
studies  more  usable  across  the  services  based  on  more  consistent 
experimental  methodologies.  Another  goal  is  to  provide  a  mechanism  for 
integrating  and  interpreting  the  individual  contributions  being  made  to 
APM  by  various  researchers  and  organizations.  These  goals  take  on 
additional  meaning  when  it  is  considered  that  recent  analyses  (e.g. 
Cotton,  1978)  show  that  measurements  previously  possible  only  in 
simulators  can  be  made  in  flight  using  airborne  computers  which  are 
hardware  and  software  compatible  with  specific  ATDs.  Implications  for 
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highly  objective  training  effectiveness  and  operational  readiness 
assessments  are  much  encouraged  by  these  analyses. 

Automated  measurement  of  aircrew  member  performance  in  ATDs  has 
proceeded  along  logical  and  coordinated  paths*  but  only  for  a  relatively 
short  period  of  time.  Research  in  this  admittedly  complex  technical 
area  is  bearing  fruit,  which  ultimately  may  be  applied  in  flight. 

Design  Recommendations 

Guidelines  for  the  design  of  practical,  valid  and  acceptable  APM 
systems  are  not  yet  at  hand.  Human  performance  is  complex  and  the 
human's  evaluation  of  another  human's  performance  also  is  complex. 
Computer-based  systems  for  assessing  and  diagnosing  human  performance 
just  are  beginning  to  evolve.  Operational  applications  of  true  APM 
systems  basically  are  non-existent. 

APM  system  design  recommendations  are,  thus,  both  simple  and 
complex.  First,  candidate  measures  must  be  determined  both  on  the  basis 
of  existing  operational  doctrine  (which  often  reflects  the  limitations 
of  human  performance  measuring  capabilities),  and  detailed  measurement 
analyses  and  subsequent  measurement  development  and  validation 
experiments. 

The  design,  development  and  validation  of  workable  automated 
performance  measurement  systems  for  aircrew  training  requires  an  orderly 
analytic  and  experimental  process.  Discussions  with  a  respected  APM 
researcher  revealed  that  he  would  not  participate  in  developing  an  APM 
specification  unless  the  specification  included  the  requirement  for  a 
formal  development  and  validation  experimental  program.  This  need  will 
continue  until  enough  experimental  and  operation  experience  is  gained. 
These  requirements  cannot  now  be  met.  Appendix  B  was  prepared  to 
present  guidance  on  these  and  related  issues.  The  interested  reader  is 
referred  to  the  appendix  for  a  more  detailed  discussion  of  APM  system 
design  and  validation. 

AUTOMATED  PERFORMANCE  ALERTS 

Summary 

Automated  performance  alerts  are  signals  provided  to  the  instructor 
and/or  the  pilot  being  trained  indicating  that  some  facet  of  his 
performance  has  exceeded  acceptable  limits.  Automated  alerting 
capabilities  require  an  automated  performance  measurement  system.  Their 
use  also  requires  that  training  tasks  be  specified  and  standardized 
sufficiently  that  automated  performance  measurement  is  possible.  Only 
one  ATD  (a  helicopter  OFT)  was  observed  during  program  site  visits  that 
Incorporated  any  form  of  automated  performance  alert.  This  system 
produced  a  symbol  on  the  instructor's  electronically  generated  flight 
situation  (graphic  position)  display  at  points  along  the  flight  profile 
where  predetermined  performance  limits  were  exceeded.  Results  of  a 
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prior  training  effectiveness  evaluation  of  a  navigator  training 
simulator  showed  that  performance  alerts  may  be  disruptive  to  learning 
if  they  are  presented  directly  to  the  student  because  of  the  distraction 
they  can  cause.  To  be  acceptable  to  users,  automated  performance 
alerting  capabilities  must  be  designed  so  that  the  alerts  are  neither 
excessive  in  number  nor  disruptive  to  student  or  Instructor  activities. 
If  an  alerting  capability  is  incorporated  into  an  ATD,  the  ability  to 
turn  the  capability  on  or  off  at  the  user's  discretion  also  should  be 
provided. 

Feature  Description 

Automated  performance  alert  signals  are  intended  to  enhance  the 
monitoring  of  student  performance  by  combining  the  capabilities  of  an 
automated  performance  measurement  system  with  an  automated  capability 
intended  to  complement  the  Instructor  (or  student's)  normal  monitoring 
of  performance  information  provided  to  him. 

The  use  of  automated  measurement  requires  that  the  training  tasks 
can  be  specified  and  standardized  sufficiently  so  that  automated 
measurement  Is  possible.  For  example,  the  use  of  automated  alerts  in 
the  training  of  many  flight  control  and  navigation  tasks  requires  the 
measurement  system  to  know  which  task  is  being  performed  by  the  student 
so  that  appropriate  measurement  criteria  can  be  applied  for  the  alerting 
function.  One  method  of  doing  this  is  through  the  use  of  pre-progranmed 
mission  scenarios  (See  Chapter  III).  Some  sophisticated  laboratory 
automated  measurement  systems  (e.g.  Vreuls  et  al.,  1975)  use  complex 
computer  logics  to  identify  the  task  the  student  is  performing  by 
monitoring  the  ATO's  flight  path,  various  flight  parameters  and  the 
positions  of  relevant  controls  such  as  navigation  radio  control 
settings.  Until  recently,  such  logics  existed  only  for  a  limited 
spectrum  of  instrument  flight  tasks.  The  Navy  is  sponsoring  the 
development  and  evaluation  of  similar  logics  for  determining  when 
procedural  sequences  (e.g.  engine  start  and  before  takeoff  checklists) 
have  been  initiated  and  completed.  Pursuit  of  these  initial 
developments  will  expand  automated  performance  measurement  and  alerting 
Into  the  procedures  monitoring  area.  Finally,  some  performance  alerts 
are  relatively  easy  to  generate  because  they  are  largely  Independent  of 
specific  tasks  being  trained.  Examples  Include  alerts  when  airspeed 
exceeds  250  knots  below  10,000  feet  of  altitude,  and  when  the  landing 
gear  is  lowered  above  the  placard  airspeed. 


Alert  signals  can  be  presented  In  a  number  of  ways.  Two  common  ways 
are:  1)  visually  at  an  Instructor/operator  console  or  on  a  special 
display  In  the  ATD  student  station;  or  2)  aurally  by  means  of  a  tone. 
Verbal  alerting  messages  using  computer  generated  speech  technology  also 
are  within  the  state  of  the  art.  (See  Chapter  VI  for  a  discussion  of 
computer  generated  speech.) 


Alerts  were  presented  visually  in  the  ATDs  reviewed  in  this  study. 
One  ATD  was  Amy  rotary  wing  training  device  2B24.  The  second  was  Navy 
communication  and  navigation  training  device  1D23.  In  device  2824, 
unique  symbols  were  displayed  on  an  instructor's  electronically 
generated  flight  situation  (graphic  position)  display.  Device  ID23 
displayed  a  brief  text  message  directly  to  the  student  on  a 
special-purpose  display  (Semple,  1974).  The  student  was  required  to 
depress  a  button  within  a  short  time  period  after  the  message  was 
displayed  in  order  to  acknowledge  it.  If  he  did  not,  a  tone  sounded. 

Related  Support  Features 

Automated  performance  alerts  are  related  to  the  following 
instructional  support  feature: 

Automated  Performance  Measurement.  An  automated  performance 
measurement  system  is  required  to  activate  automated  performance 
alerts.  (See  the  first  section  of  the  chapter.) 

Instructional  Values 


The  most  commonly  advanced  instructional  value  is  an  automated 
assist  to  the  instructor  in  monitoring  the  many  channels  of  information 
that  describe  student  performance,  and  in  determining  whether 
performance  is  within  acceptable  limits  on  each  channel.  These  are  not 
tasks  at  which  the  human  excels.  Computers,  on  the  other  hand,  are  well 
suited  to  monitoring  and  evaluating  many  different  channels  at  the  same 
time. 


Properly  designed,  automated  performance  alerting  capabilities  can 
provide  feedback  to  the  pilot  being  trained  without  the  presence  of  an 
instructor.  However,  as  discussed  previously,  excessive  alerts  can  be 
distracting  and  disruptive  to  the  learning  process.  Also,  alerts  used 
in  this  way  must  specifically  identify  the  student's  performance  problem 
(e.g.  below  glidepath)  rather  than  just  indicate  "you've  done  something 
wrong." 

Observed  Applications 

The  only  application  of  automated  performance  alerts  observed  during 
this  program  was  in  Device  2824,  the  Amy  UH-1  helicopter  operational 
flight  trainer.  In  this  application,  performance  alerts  were  displayed 
visually  as  unique  symbols  beside  a  flight  path  trace  on  an 
electronically  generated  graphic  position  display  at  the 
Instructor/operator's  console.  The  performance  alert  symbols  could  be 
displayed  or  deleted  at  the  Instructor's  discretion. 

Utility  Information 


The  design  of  the  automated  performance  alerting  capability  of 
Device  2824  allowed  Instructors  to  select  whether  or  not  they  wanted  the 


alert  symbols  displayed.  This  capability  was  viewed  favorably  by 
Instructors.  The  symbols  could  be  deleted  to  minimize  display  clutter. 
Later,  following  the  flight,  the  history  of  the  flight  could  be 
displayed  together  with  the  alert  symbols  as  a  debriefing  aid.  This 
provided  a  well  accepted  memory  aid  as  well  as  a  pictorial  Index  of 
student  performance.  For  example,  fewer  performance  alert  symbols  on 
the  display  Indicated  better  performance.  Use  of  the  display  for 
debriefing  was  limited,  however,  when  ATD  utilization  was  high.  A  hard 
copy  capability  would  have  alleviated  this  problem. 

Performance  alerts  can  be  distracting.  Semple  (1974)  reported  that 
an  alerting  capability  built  into  a  Navy  undergraduate  navigator  trainer 
was  deactivated  after  a  short  period  of  trial  use  because  it  was  found 
to  be  disruptive  to  the  learning  process.  In  this  case,  students  having 
difficulty  performing  navigation  tasks  were  sufficiently  busy,  that 
having  to  respond  to  a  performance  alert  only  aggravated  their 
situation.  It  is  possible  that  voice  alerts  might  not  have  been  as 
disruptive.  However,  frequently  occurrir;$»  voice  alerts  also  may  be 
aggravating.  Instructors  involved  in  the  study  conmented,  however,  that 
such  an  automated  performance  alert  capability  at  their  remote  consoles 
could  have  been  of  value  because  each  instructor/operator  team  had  to 
monitor  the  performance  of  up  to  20  different  students  and  share  their 
time  among  students  who  needed  help  the  most. 

Related  Research 

No  systematic  research  on  effective  designs  and  uses  of  automated 
performance  alerts  for  training  Is  known  to  exist.  Such  research  should 
be  undertaken  to  define  the  applications  and  limits  of  this  potentially 
useful  Instructional  tool.  The  only  other  known  aircrew  training  study 
involving  automated  performance  alerts  (Semple,  1974)  has  been  discussed 
in  preceding  parts  of  this  section. 

Design  Considerations 

Research  is  needed  on  many  automated  alerting  design  guide  issues. 
The  following  design  considerations  are  provided  In  part  to  guide  the 
needed  research. 

It  appears  logical  to  design  automated  alerting  systems  with'  at 
least  three  and  possibly  more  different  sets  of  performance  standards. 
Early  In  the  learning  process,  many  mistakes  are  made.  Indeed,  the 
process  of  training  Is,  In  part,  designed  to  eliminate  mistakes  by 
directing  performance  toward  acceptable  standards.  An  alerting  system 
that  closely  monitors  student  performance  early  in  training  according  to 
standards  expected  at  the  conclusion  of  training  surely  will  Inundate 
the  Instructor  or  student  with  performance  alerts.  Therefore,  the  use 
of  performance  standards  that  are  appropriate  for  the  student's  level  of 
performance  during  the  learning  process  seems  more  reasonable.  This 
might  require  up  to  three  and  possibly  more  sets  of  performance 
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standards.  Developmental  research  on  performance  standards  should  be  an 
integral  part  of  the  design  of  future  performance  alerting  systems. 

Any  performance  alert  should  be  designed  with  feedback  and  guidance 
goals  in  mind.  An  alert  that  simply  says  "something  is  wrong"  tells 
neither  the  instructor  nor  the  student  what  is  wrong.  Without  specific 
feedback,  specific  guidance  cannot  be  given  the  student  on  what  facet  of 
his  performance  he  should  change.  This  suggests  an  alerting  system  with 
at  least  two  levels  of  feedback.  The  most  basic  levels  would  be  simply 
to  highlight  a  performance  deficiency.  The  second  level  should 
specifically  identify  the  deficiency  so  that  feedback  and  guidance  can 
be  specific.  On  the  other  hand,  too  much  information  can  be  as  useless 
as  too  little.  Further  research  is  required  to  develop  meaningful 
guidelines  on  the  amount  and  timing  of  automated  performance  alert 
information. 

Computer  speech  generation  technology  presently  is  available  as 
a  means  of  providing  performance  alert  information.  This  is  a  "natural" 
communication  medium  and  one  that  does  not  require  the  student  or 
instructor  to  monitor  special  displays  providing  alert  information. 
However,  this  medium  is  untried  as  an  alerting  channel.  Again,  research 
is  required. 

ANNUNCIATORS  AND  REPEATER  INSTRUMENTS 
Summary 

Repeater  flight  instruments  and  annunciators  that  signal  the 
occurrence  of  an  event  (such  as  the  change  in  the  positioning  of  a 
control)  are  issues  that  are  relevant  mostly  for  instructor  consoles 
that  are  remote,  where  the  instructor  cannot  directly  observe  student 
actions  and  the  displays  and  controls  that  are  at  the  student's  training 
station.  The  goal  is  to  provide  the  instructor  with  information 
relevant  to  his  instructing  the  student  and  evaluating  student  actions 
for  the  purpose  of  providing  feedback  and  guidance.  Many  remote 
instructor  consoles  incorporate  repeater  (actual  replica)  flight 
instruments.  Electronic  display  technology  makes  it  possible  to  create 
symbolic  representations  of  these  displays,  and  this  is  being  done  more 
frequently.  Annunciators  (usually  illuminated  Indicators)  have  been 
used  in  the  past  to  inform  the  instructor  of  the  positions  of 
significant  controls.  Modem  technology  makes  It  possible  to  present 
this  Information  on  electronically  generated  displays.  Significant 
questions  remain  regarding  the  best  combinations  of  design  possibilities 
to  provide  the  Instructor  with  the  flight,  control  and  event  Information 
required  to  Instruct  the  student  and  evaluate  his  performance. 

Organization  of  the  content  of  this  section  departs  from  the  format 
previously  used  because  of  the  nature  of  the  Information  that  can  be 
provided.  This  topic  Is  directly  related  to  Instructor/operator  console 
design;  therefore,  the  reader  Is  referred  to  Chapter  VIII  of  this 
report. 
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Annunciator  Displays 

Annunciator  displays  come  in  many  forms.  Some  are  flap  or  wing 
sweep  mechanical  indicators.  Others  are  discrete  event  indicators. 
Usually,  these  are  illuminated  indicators  that  light  up  when  an  event 
occurs,  such  as:  gear  up;  bombing  mode  control  set  to  ripple  bomb;  or 
head  up  display  set  to  attack  mode.  In  each  case,  the  indicator  at  a 
remote  instructor/operator  console  is  intended  to  inform  the  instructor 
of  critical  control  positions  selected  by  the  pilot  being  trained. 
Several  shortfalls  have  been  observed  during  this  program  regarding  the 
design  of  annunciator  displays.  Each  is  summarized  below. 

The  information  provided  often  is  incomplete.  Since  there  is 
nothing  sacred  about  the  skin  of  an  ATO,  an  alternative  worth 
investigating  is  the  provision  for  direct  instructor  observation  of 
student  performance,  particularly  during  phases  of  training  where 
procedures  and  control  positioning  are  important  training 
objectives.  The  value  of  this  alternative  hinges,  of  course,  on  all 
relevant  controls  and  displays  being  visible  to  the  instructor. 

The  information  provided  can  be  overwhelming.  Too  many  annunciator 
displays  simply  can  overwhelm  the  fundamental  human  ability  to 
monitor  and  process  information.  The  situation  can  be  fully 
aggravated  when  one  stops  to  consider  the  hundreds  of  control 
positions  that  the  instructor  needs  to  be  aware  of.  The  problem  is 
particularly  acute  when  the  information  is  provided  in  formats  and 
arrangements  that  are  not  operationally  meaningful  to  the 
instructors.  Automated  performance  monitoring  could  reduce  this 
problem.  (See  the  previous  section  of  this  chapter.) 

The  arrangement  of  annunciators  is  illogical.  Often,  annunciators 
at  an  instructor  console  are  arranged  according  to  the  sequence  in 
which  the  ATO  design  engineer  thinks  they  will  be  used.  This 
sequence  often  bears  little  or  no  relationship  to  the  operational 
sequences  that  instructors  are  used  to  looking  for. 

The  current  trend  is  toward  electronic  display  of  information  at 
instructor/operator  consoles.  This  means  that  display  pages  easily 
could  be  created  showing  lists  of  control  events  performed.  However, 
this  too  could  result  In  an  information  overload,  which  would  be 
similar  to  the  second  shortcoming  listed  above.  Additional  research  is 
required  on  how  to  effectively  display  discrete  event  procedures 
Information  (and  continuous  data  showing  trend  information)  on 
electronic  displays.  One  option  Is  to  relegate  the  task  of  performed 
monitoring  to  an  automated  performance  measurement  system,  and  only 
display  departures  from  accepted  procedural  sequences.  However,  the 
matter  of  information  display  still  requires  examination. 
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Repeater  Instruments 

Repeater  Instruments  displaying  flight  control  information  have  been 
used  at  remote  instructor  consoles  for  many  years  to  provide  pilot 
instructors  with  performance  data  in  formats  they  are  used  to.  Present 
technology  allows  for  the  presentation  of  such  information  using 
electronically  generated  displays,  rather  than  actual  cockpit 
instruments.  Advantages  lie  in  the  area  of  cost  and  maintainability. 
Potential  disadvantages  lie  in  providing  information  in  formats  that 
instructor  pilots  are  not  highly  familiar  with.  Only  one  ATD  (F-15  OFT) 
was  reviewed  during  this  program  that  used  electroncially  generated 
flight  displays.  The  F-15  instructor  pilots  interviewed  reported  that 
the  displays  were  natural  to  interpret  and  use  to  monitor  flight 
performance. 

CLOSED  CIRCUIT  TELEVISION 
Summary 

Closed  circuit  television  (CCTV)  systems  have  been  designed  into 
several  training  and  research  simulators  on  the  premise  that  student 
behavior  could  be  monitored  better  from  a  remote  instructor  console  or 
that  improved  debriefing  information  could  be  made  available.  None  of 
the  sytems  installed  to  date  has  proved  workable  due  to  poor  image 
quality,  narrow  field  of  view,  or  poor  viewing  angles. 

Feature  Description 

CCTV  systems  incorporated  a  camera,  a  TV  monitor  for  real  time 
viewing,  and,  depending  on  their  design  and  intended  use,  a  video  tape 
system  for  post-mission  replay.  Systems  installed  to  date  have  been 
black  and  white.  Most  cameras  have  been  positioned  to  "look  over  the 
student's  shoulder",  and  provide  images  of  the  student,  portions  of  the 
forward  instrument  panel,  and  portions  of  the  visual  scene  for  visually 
equipped  ATDs.  One  was  designed  to  video  tape  scenes  of  instructor 
console  displays  for  debriefing  use.  This  was  in  the  Air  Force 
Simulator  for  Air  to  Air  Combat  (SAAC.) 

Related  Instructional  Support  Features 

CCTV  is  unrelated  to  other  support  features. 

Instructional  Values 


Assumed  instructional  values  fall  into  two  categories.  The  first 
assumed  value  Is  providing  improved  monitoring  of  student  bahavior  in 
ATDs  that  have  been  designed  so  that  direct  instructor  observation  is 
not  practical.  The  second  assumed  value  is  providing  real-time 
debriefing  information. 
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Observed  Applications 

Four  applications  of  CCTV  were  identified  during  the  program.  The 
prototype  of  Device  2B24,  used  for  initial  entry  training  of  UH-1 
helicopter  pilots,  included  a  CCTV  system.  Production  units  did  not 
because  of  the  inadequate  field  of  view.  The  Air  Force  Advanced 
Simulator  for  Pilot  Training  (ASPT)  incorporates  an  over  the  shoulder 
CCTV  system.  The  Air  Force  SAAC  device  incorporates  a  CCTV  system  to 
record  display  information  at  the  instructor  console  for  later  replay 
during  debriefing.  The  Northrop  Corporation's  Large  Amplitude 
Simulator/Wide  Angle  Visual  System  (LAS/WAVS)  device  incorporated  an 
over  the  shoulder  CCTV  system  to  allow  monitoring  of  an  adversary 
aircraft  and  pilot  actions  during  one  versus  one  air  combat  maneuvering. 

Utility  Information 

None  of  the  CCTV  systems  identified  during  STRES  was  found  to  be 
usable.  Poor  image  quality  was  one  reason.  Restricted  field  of  view 
was  another.  And,  over  the  shoulder  views  mostly  show  the  student's 
back,  and  little  of  what  he  is  looking  at  or  doing.  Although  image 
quality  and  field  of  view  problems  can  be  overcome,  camera  location 
problems  may  not  be  easily  resolved. 

Design  Considerations 

To  date,  no  ATD  has  incorporated  a  CCTV  system  that  was  designed  to 
provide  a  front  view  of  the  student,  rather  than  an  over  the  shoulder 
view.  A  front  view  system  might  provide  relevant  training  information 
on  pilot  visual  scan  patterns  and/or  hand  movements.  The  possibility 
exists  that  such  a  system  could  interfere  with  a  simulated  out  of  the 
cockpit  visual  scene  presentation.  However,  fiber  optics  technology 
offers  a  means  around  this  potential  problem.  The  training  value  of 
this  type  of  system  is  an  experimental  question. 
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CHAPTER  V 


CONTROL  AND  INDIVIDUALIZE  TRAINING 


FUNCTION  DEFINITION 

Control  means  to  direct  or  regulate  training  events. 
Individualizing  training  is  a  special  case  of  controlling,  where 
training  is  tailored  to  the  student's  specific  skill  levels  and 
learning  needs. 

RELATED  FUNCTIONS 

The  reader  is  directed  to  the  chapters  of  this  report  listed  below 
that  address  functions  related  to  controlling  and  individualizing 
training.  The  reader  also  is  referred  to  Chapter  IV  of  the  utilization 
report  for  related  information  on  the  structuring  of  ATD  training  and 
assessing  training  effectiveness. 

Chapter  IV,  Monitor  and  Evaluate  Performance 

Chapter  VII,  Prepare,  Brief  and  Debrief 
INSTRUCTIONAL  SUPPORT  FEATURES  ADDRESSED 

Instructional  support  features  addressed  in  this  chapter  are: 

Automated  adaptive  training 

Programmed  mission  scenarios 
AUTOMATED  ADAPTIVE  TRAINING 
Summary 

Automated  adaptive  training  is  a  complex  and  developing  technology 
in  which  instructional  objectives  and  strategies  that  are  implemented  as 
computer  software  are  used  to  automatically  tailor  (adapt)  instruction 
to  Individual  student  learning  needs.  The  present  status  of  automated 
adaptive  training  is  reflected  in  the  fact  that  there  appears  to  be  no 
well  established,  commonly  used  definition  of  the  concept  with 
meaningful  Implications  for  how  adaptive  training  systems  should  be 
conceived  and  designed.  It  generally  Is  assumed  that  automated  adaptive 
training  systems  will  hold  the  potential  to  achieve  very  efficient, 
highly  individualized  training  while  ensuring  that  all  students  also 
achieve  specified,  measurable  skill  levels  at  the  conclusion  of 
training.  Present  research,  which  has  been  limited  and  exploratory  in 
nature,  has  Involved  only  a  sampling  of  Instructional  strategies  and 
measurement  methods,  and  has  Involved  only  limited  Instrument  flight 
training  tasks.  Research  conducted  to  date  has  not  clearly  demonstrated 
any  training  advantages  to  automated  adaptive  training.  However, 
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available  findings  do  suggest  potentially  fruitful  avenues  for  future 
research  in  this  new  and  complex  area. 

Feature  Description 

In  a  general  sense,  automated  adaptive  training  (AAT)  is  the 
computer  implementation  of  learning  strategies  and  objectives  with  the 
general  goals  of  individualizing  aircrew  member  training,  achieving 
training  efficiencies  by  focusing  instruction  on  student  performance 
deficiencies,  and  ensuring  that  all  students  also  achieve  specified, 
measurable  skill  levels  at  the  end  of  training.  AAT  is  defined  in  this 
report  as  training  in  which  cues  or  tasks  are  varied  automatically  as  a 
function  of  how  well  the  student  performs.  This  definition  is  adapted 
from  the  definition  of  AAT  used  by  Kelley  (1969).  Hughes  (1979)  defines 
AAT  as  a  technique  in  which  the  complexity  and/or  difficulty  of  a  task 
is  adapted  automatically  to  the  skill  level  of  the  student.  Still 
others  (e.g.  Wooldridge,  Vreuls  and  Norman,  1977),  who  have  done 
research  on  AAT  for  aircrew  training,  offer  no  specific  definition  of 
the  term,  perhaps  because  they  hoped  the  results  of  their  work  would 
provide  a  more  empirical  basis  for  defining  the  concept  for  aircrew 
training  applications. 

The  adaptation  of  learning  experiences  to  student  needs  probably  was 
used  by  Socrates.  Skilled  instructors,  today,  tailor  training  to 
student  needs.  It  is  attempts  to  automate  the  adaptation  of  training 
that  are  new.  Essentially,  therefore,  AAT  systems  reauire  the  automation 
of  instructor  models.  Key  questions  are:  1)  what  instructor 

(instructional)  strategies  and  decisions  should  be  automated;  2)  how 
must  these  be  tailored  based  on  differences  in  student  learning  styles 
(Sullivan,  Casey  and  Hebein,  1978);  3)  what  facets  of  student 
performance  must  be  monitored  and  evaluated  so  that  student  performance 
strengths  and  learning  deficiencies  can  be  objectively  and  reliably 
identified,  so  that  subsequent  training  can  be  based  on  them;  and  4)  how 
are  automated  instructor  models  impacted  by  the  tasks  being  trained  and 
whether  they  are  individual  crewmember  tasks  or  team  tasks?  These  are 
complex  issues  and  questions.  For  example,  it  does  not  appear  totally 
clear  how  to  train  (program)  human  instructors  to  be  effective, 
efficient  teachers  of  aircrew  members,  at  least  based  on  program 
surveys.  Additionally,  military  missions  require  that  aircrew  members 
be  able  to  perform  a  very  wide  variety  of  tasks.-  Also,  AAT  systems 
require  highly  sophisticated  automated  performance  measurement 
capabilities  (e.g.  Wooldridge,  et  al.,  1977),  which  also  are  at  the 
forefront  of  technology.  It  should  be  no  surprise,  therefore,  that  AAT 
concepts,  definitions,  research  and  initial  findings  are  purely  at  the 
exploratory  development  level,  and  are  not  yet  ready  for  application  to 
a  majority  of  day-to-day  training  needs  (McCauley  and  Semple,  1980). 

Related  Support  Features 


AAT  is  directly  related  to  automated  performance  measurement. 
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Automated  Performance  Measurement  (APM).  Automated  measurement  is 
necessary  for  AaT  systems  to  determine  how  well  the  student  is 
performing  and  to  identify  specific  performance  strengths  and 
deficiencies  so  that  subsequent  training  can  be  tailored 
automatically  to  the  student's  needs.  It  is  believed  that  AAT 
systems  for  aircrew  training  will  require  highly  sophisticated 
automated  measurement  capabilities.  (See  Appendix  B  of  this 
report . ) 

Instructional  Values 


It  is  assumed  that  automated  adaptive  training  systems  hold  the 
potential  to  achieve  very  efficient,  highly  individualized  training 
while  ensuring  that  all  students  also  achieve  specified,  measurable 
skill  levels  at  the  conclusion  of  training  (Conway  and  Norman,  1974). 

Observed  Applications 

Only  one  system  was  observed  during  program  surveys  that 
Incorporated  a  capability  even  approaching  AAT.  This  was  Army  training 
device  2B24.  At  the  Instructor's  discretion,  the  device  could  be 
conmanded  to  automatically  increase  turbulence  and  crosswinds  during 
instrument  approach  and  departure  training  depending  on  how  well  the 
student  had  performed  the  previous  attempt  at  the  task.  This  feature  of 
the  device  was  used  at  the  Instructor's  discretion,  and  no  indications 
could  be  obtained  on  its  perceived  value.  Recent  research  (Wooldridge, 
Vreuls  and  Norman,  1977)  suggests  that  such  “problem  difficulty" 
adjustments  during  initial  task  learning  may  interfere  with  the  learning 
process. 

Utility  Information 

No  practical  utility  information  is  available  because  no 
true  applications  of  AAT  were  observed  during  this  program. 

Related  Research 


There  has  been  very  little  experimental  research  dealing  with  AAT 
for  aircrew  training.  Several  preliminary  AAT  systems  were  placed  in 
the  field  for  evaluation  at  NAS  Chase  Field  and  Luke  AFB.  The  systems 
were  designed  for  AAT  of  the  GCA  maneuver.  Although  the  systems 
provided  well  accepted  features  for  training  task  setup,  control  and 
performance  feedback,  problems  with  the  efficiency  of  training  that 
could  be  accomplished  with  the  systems  were  observed  (Puig  and  Gill, 
1975;  and  Brown,  Waag  and  Eddowes,  1975).  The  design  of  these  systems 
was  based  on  concepts  developed  for  feasibility  demonstration  only 
(Charles  and  Johnson,  1971;  Charles,  Johnson  and  Swink,  1972;  and 
Charles,  Johnson  and  Swink,  1973).  Practically  none  of  the  design 
Issues  had  been  researched.  Rather,  design  decisions  were  based  on  "best 
estimates"  of  training  and  software  specialists.  The  resulting  systems 
demonstrated  a  working  application  of  AAT  and  were  shown  to  have  some 
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appeal  based  on  pilot  opinion  (Wooldridge,  Vreuls  and  Norman,  1977).  No 
claims  were  made,  however,  that  the  AAT  system  designs  were  either 
optimum  or  efficient. 

The  complexity  of  AAT  design  and  use  issues  is  reflected  in  a 
recently  reported  study  (Wooldridge,  et  al.,  1977),  which  investigated 
several  different  experimental  adaptive  training  logics  for  use  in 
training  relatively  naive  pilots  to  perform  basic  instrument  flight 
maneuvers  (straight  and  level  flight,  climbs  and  dives,  level  turns, 
climbing  and  diving  turns,  and  GCA).  Aircraft  weight,  center  of 
gravity,  turbulence  and  steady  state  winds  (crosswinds)  also  were 
varied.  In  addition  to  the  AAT  logics  studied,  they  also  included  two 
control  groups  of  interest.  One  control  group  received  task  training  in 
a  random  sequence  (versus  according  to  a  particular  adaptive  training 
logic).  Two  instructors,  working  from  a  remote  console,  also  provided 
"manually  controlled"  training  as  a  second  control  condition. 

The  adaptive  training  logics  studied  by  Wooldridge  et  al.  were 
rather  complex  and  were  intended  to  sample  the  thinking  of  training 
researchers  at  the  time.  Without  pursuing  the  details  of  the  logics 
they  studied,  several  findings  are  important.  First,  in  terms  of  the 
amount  of  time  required  for  students  to  complete  the  experimental 
syllabus,  the  random  presentation  of  tasks  proved  to  be  one  of  the  best 
instructional  strategies.  Second,  none  of  the  automated  instructional 
strategies  was  as  efficient  as  of  one  of  the  instructors,  who  apparently 
disregarded  variations  in  task  difficulty  (e.g.  turbulence,  gross 
weight,  etc.)  until  his  students  had  mastered  the  basic  tasks.  Then  he 
exposed  them  to  the  various  "difficulty"  factors. 

Results  of  the  study  by  Wooldridge,  et  al.  are  somewhat  difficult  to 
Interpret  because  they  suggest  that  not  all  student  groups  were  trained 
to  the  same  levels  of  proficiency.  Thus,  comparing  group  performances 
based  on  training  times  Is  difficult  at  best.  However,  the  fact  that 
either  a  good  instructor  or  a  random  sequence  of  training  events  could 
result  in  improved  training  efficiencies  (compared  with  the  adaptive 
logics  studied)  Indicates  that  AAT  technology  is  not  yet  ready  for 
operational  application,  whether  the  training  tasks  involve  basic 
Instrument  flight  maneuvers  or  more  complex,  operationally-oriented 
tasks.  Further  research  Is  needed  in  this  potentially  productive  area. 

Design  Recommendations 

No  AAT  system  design  recommendations  can  be  made  based  on  available 
operational  or  experimental  evidence.  AAT  Is  an  area  requiring 
considerable  future  research. 


PROGRAMMED  MISSION  SCENARIOS 
Summary 

Programmed  mission  scenarios  can  be  thought  of  as  non-adaptive 
training  exercises*  Programmed  scenarios  are  not  modified  to  reflect 
changes  in  student  performance.  Rather,  they  consist  of  highly 
structured  sets  of  events  that  often  are  patterned  after  the  content  of 
existing  blocks  of  instruction.  At  least  one  developmental  system  (an 
experimental  prototype  Navy  F-14  OFT  instructional  support  system) 
incorporates  a  shopping  list  of  task  modules  that  instructors  can  use  to 
create  special  programmed  missions  that  can  be  tailored  to  specific 
student  needs.  The  programed  mission  scenario  feature  was  used  a  great 
deal  in  one  SAC  mission  trainer  for  training  involving  penetration 
through  known  high  threat  environments.  In  this  application,  various 
threats  were  pre-programed  to  occur  at  meaningful  points  in  the 
mission. 

Feature  Description 

Programed  mission  scenarios  are  highly  structured,  programed 
sequences  of  events.  Factors  that  can  be  programed  include:  flight 
tasks  (mission  segments);  malfunctions;  threats;  and  environmental 
variables.  Usually,  a  specific  chock  to  chock  mission  profile  is  used. 
However,  individual  segments  of  a  total  mission  or  specific  tasks  or 
maneuvers  can  be  used  (Hughes,  1979). 

Related  Support  Features 

Programed  mission  scenarios  are  most  directly  related  to  the  five 
instructional  features  identified  below: 

Recorded  briefings.  Recorded  briefings  provide  a  convenient 
briefing  tool  for  missions  that  are  programed.  (See  Chapter  VII) 

Automated  controllers.  Automated  voice  controllers  can  be  used  with 
programed  mission  scenarios,  as  can  "iron  pilots"  to  control 
adversary  aircraft.  (See  Chapter  VI) 

Freeze.  Total  system  freeze  and  flight  system  freeze  can  be  used 
meaningfully  during  programed  missions.  (See  Chapter  III) 

Automated  performance  measurement.  Automated  measurement  requires 
that  a  specifiable  set  of  tasks  and  task  sequences  be  used  as  the 
basis  for  measuring.  Programed  missions  are  one  means  of 
providing  specifiable  tasks  and  sequences.  (See  Chapter  IV) 

Assumed  Values 


Two  Insructional  values  of  programed  mission  scenarios  commonlv  are 
assumed.  The  first  is  standardization  of  trailing,  or  at  least 
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standardization  of  practice  opportunities.  All  training,  of  course, 
cannot  not  be  relegated  to  programed  event  sequences,  since  doing  so 
would  ignore  the  need  to  focus  much  of  training  on  specific  student 
learning  needs.  Nonetheless,  programed  mission  scenarios  provide 
opportunities  for  the  standardization  of  at  least  some  facets  of  aircrew 
training.  An  example  is  posing  realistic  threat  problems  to  aircrew 
members  in  a  context  that  will  allow  them  to  practice  dealing  with  the 
threats  until  acceptable  performance  is  demonstrated. 

Programmed  missions  can  provide  opportunities  for  automated 
performance  measurement  and  student  evaluation.  This  follows  since  the 
baseline  of  performance  often  is  defined  by  a  highly  structured 
scenario.  A  full  mission  scenario,  however,  is  not  required  for 
automated  measurement.  The  only  requirement  is  that  the  measurement 
system  must  "know",  unambiguously,  what  task  is  being  performed  so  that 
it  can  apply  the  appropriate  measurement  algorithms.  A  programmed 
scenario  is  one  way  of  informing  the  measurement  system  of  what  tasks 
are  being  performed. 

A  potential  value  of  programmed  scer’rios  is  to  enable  structured 
practice  opportunities  for  students  who  may  be  using  an  ATD  without  an 
instructor  present.  In  these  cases,  a  device  operator  could  call 
up/ load  the  programmed  mission  requested  by  the  student,  and  the  student 
then  could  practice  the  tasks  required  in  the  mission. 

Observed  Applications 

Only  one  application  of  programmed  mission  scenarios  was  observed 
during  program  site  visits.  This  was  on  FB-111  simulation  that  was  used 
for  transition  and  continuation  training.  The  application  involved 
training  on  penetration  through  high  threat  environments.  The  mission 
scenarios  were  highly  realistic  and  Involved  known  and  anticipated 
threats  that  aircrew  members  would  encounter  in  an  emergency  war  order 
mission. 

Utility  Information 


Instructors  and  students  who  had  experience  with  them  viewed 
programmed  scenarios  favorably.  Programming  mission  events  greatly 
simplified  Instructor  activities  at  the  ATE)  console  because  they  did 
not  have  to  manually  control  each  event.  Rather,  they  were  able  to 
devote  their  time  to  monitoring  student  performance  and  providing  verbal 
instruction.  Students  responded  favorably  for  the  same  reasons. 
Several  students  also  pointed  out  that  meaningful  threats  were  not 
accidentally  excluded  due  to  instructor  workload  or  diversion. 

Many  instructors  and  students  that  have  not  had  experience  with 
programmed  missions  are  apprehensive  about  them.  Their  apprehensions 
stem  from  rigidity  associated  with  programmed  events.  Their  concern 
centers  on  the  possibility  that  the  situations  and  cues  that  students 
would  experience  would  be  too  restricted  and  could  lead  to  safety 
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problems  during  actual  flight.  Of  course,  the  validity  of  this  concern 
hinges  directly  on  how  the  programmed  scenarios  are  designed  and  used. 
This  apprehension  must  be  acknowledged.  However,  it  also  must  be 
acknowledged  that  relatively  few  instructors  or  students  have  actual 
experience  with  programmed  missions. 

Related  Research 


The  Navy  is  developing  an  experimental  prototype  instructional 
support  system  that  will  complement  an  existing  i nstructor/operator 
console  on  an  F-14  operational  flight  trainer.  The  Air  Force  is 
developing  a  similar  system  for  the  C-5A  ATD.  Both  systems  are 
experimental  prototypes,  and  will  not  be  operational  until  at  least 
1980. 

The  Navy  system  was  designed  to  meet  a  number  of  objectives,  which 
include:  automated  training  involving  programed  mission  scenarios; 
adaptive  training;  and  automated  measurement  of  student  performance 
(Semple,  et  al.,‘  1979).  The  approach  taken  was  to  develop  basic  sets  of 
task  modules  which  could  be  used  as  building  blocks  to  construct 
programmed  missions,  or  create  highly  tailored  adaptive  training 
exercises.  System  modes  allow  instructors  to  select  from  pre-programmed 
missions  .(pre- programed  sequences  of  task  modules),  or  to  create, 
within  some  limits,  new  task  sequences  from  the  Instructor  console.  The 
latter  capability  was  included  to  allow  tailoring  of  training  exercise 
content  while  still  taking  advantage  of  the  basic  values  of  programed 
mission  scenarios. 

Design  Considerations 

Programed  mission  scenarios  should  be  relatively  easy  to  create  and 
to  modify.  The  basic  system  design  should  acknowledge  that  training 
requirements  change,  and  It  will  be  necessary  to  modify  programed 
scenarios  accordingly.  This  should  not  be  a  difficult,  time  consuming 
task  for  operational  personnel,  and  It  should  not  require  contractor 
participation. 

Instructors  must  be  made  aware  of  programed  events  that  are  about 
to  occur.  This  Is  necessary  not  only  for  anticipation,  but  so  they  can 
override  (l.e.  Inhibit)  programed  events.  For  example.  If  a  student 
obviously  is  working  at  capacity  for  his  skill  level,  then  further 
burdening  him  with  emergencies,  threats,  or  the  like  would  have  no 
Instructional  benefit.  In  such  cases.  Instructors  must  be  provided  with 
the  option  of  overriding  pending  events. 

It  Is  desirable  that  instructors  be  provided  with  memory  aids  that 
define  the  sequence  of  mission  events  and,  when  appropriate,  the 
conditions  that  will  Initiate  the  programed  events.  This  can  be  done 
with  either  a  hard  copy  listing  of  mission  content,  or  by  displaying  a 
mission  event  listing  at  the  Instructor/operator  console. 
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CHAPTER  VI 


CONTROLLER  FUNCTION 


FUNCTION  DEFINITION 

Control  means  to  regulate  or  direct.  Instructors  and/or  ATD 
operators  often  play  the  roles  of  air  traffic  controllers,  tactical 
controllers,  or  they  control  the  actions  of  simulated  airborne  threats. 
This  chapter  addresses  Instructional  support  feature  technology 
designed  to  assist  ATD  Instructors  in  performing  the  controller 
function. 

RELATED  FUNCTIONS 

The  reader  is  directed  to  the  following  chapters  in  this  report  for 
information  related  to  the  controller  function: 


Chapter  III,  Instruct 


Chapter  IV,  Monitor  and  Evaluate  Performance 


Chapter  VIII,  Instructor /Operator  Console  Design/Location 


INSTRUCTIONAL  SUPPORT  FEATURES  ADDRESSED 

Instructional  support  features  addressed  in  this  chapter  are: 


Automated  controllers  Incorporating  computer  generated 
voice  and  automated  speech  understanding  technology 

Graphic  and  text  readouts  of  controller  information 

Computer  controlled  (Iron  pilot)  threats 

AUTOMATED  CONTROLLERS 

Summary 

The  automated  controller  is  a  relatively  new  computer  based 
technology.  The  technology  Is  founded  on  recent  advances  that  make  It 
possible  for  computer  based  systems  to  understand  at  least  limited 
amounts  of  human  speech,  and  to  generate  speech  that  can  be  understood 
by  the  human.  Automated  controller  systems  Incorporate  models  of  the 
specific  operational  situations  that  they  control.  They  also  require 
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automated  measurement  capabilities  that  relate  actual  student 
performance  to  desired  performance  so  that  controller  messages  are 
appropriate  to  the  original  incoming  message  and  the  operational 
situation.  The  combination  of  automated  speech  understanding,  situation 
recognition  and  computer  generated  speech  is  becoming  a  powerful 
instructional  support  tool  for  automated  training  (although  only  one 
operational  training  system  (the  Automated  Flight  Training  System,  AFTS) 
currently  has  these  capabilities). 

Potential  training  benefits  steming  from  this  new  technology  lie  in 
three  areas:  1)  unburdening  Instructors  and/or  ATD  operators  from 
routine  tasks  involved  in  playing  controller  roles;  2)  increasing  the 
timeliness  and  correctness  of  controller  messages;  3)  unburdening 
instructors  from  the  measurement  of  verbal  task  performance,  and 
associated  record  keeping;  and  4)  providing  a  new  medium  through  which 
students  and  ATDs  can  interact  with  each  other  in  a  highly  natural 
manner.  As  examples  of  the  fourth  point,  it  is  technically  possible  for 
the  student  to  ask  the  ATD,  "How  did  I  do  on  that  bomb  run?"  If  the 
system  has  the  necessary  performance  models  and  performance  measurement 
capabilities,  the  ATD  computer  could  respond:  "Very  well",  and  provide 
a  detailed  performance  diagnosis  if  desired.  This  technology  also  opens 
opportunities  for  very  precise,  automated  student  coaching  and  cueing. 
The  U.S.  Navy,  this  year,  has  sponsored  a  program  to  investigate  the 
full  spectrum  of  training  applications  for  automated  speech  technology. 

Present  (1980)  prototype  training  systems  which  incorporate  computer 
speech  understanding  are  limited  to  individual  word  recognition  (IWR) 
technology.  This  technology  interprets  Individual  words  or  very  short 
phrases  spoken  by  humans;  and  requires  very  precise,  stylized  speech  by 
the  human.  Also,  IWR  technology  requires  each  student  to  repeat  each 
phrase  or  word  up  to  10  times  in  a  speech  sampling  procedure  to  "train" 
the  computer  to  consistently  understand  what  was  said. 

During  the  past  year,  connected  speech  recognition  system  technology 
has  surfaced.  This  technology  allows  people  to  speak  more  naturally, 
without  the  stylization  constraints  required  by  IWR.  Also,  connected 
speech  systems  seem  easier  to  “train". 

With  respect  to  computer  .generated  speech,  present  technology  is 
quite  adequate  for  creating  words  and  sentences  that  can  be  understood 
by  the  human.  Work  continues  on  ways  to  make  the  computer  generated 
speech  sound  more  natural.  Finally,  much  of  the  technology  needed  to 
create  the  mathematical  models  and  performance  assessment  capabilities 
required  by  automated  controllers  also  exists.  However,  It  still  Is  the 
case  that  all  such  models  require  experimental  testing  and  fine  tuning. 

Feature  Description 

Automated  voice  controllers  are  new  technology  computer  based 
systems  that:  (1)  recognize  highly  structured,  short  length  statements; 
(2)~  relate  the  statements  to  specific  operational  situations;  and  (3) 
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make  responses  appropriate  to  the  original  incoming  statement  and  the 
operational  situation.  A  typical  example  involving  both  speech 
understanding  and  speech  synthesis  might  be: 

Incoming  Communication:  "Approach  Control  -  X  RAY  I  turning  to 

From  Pilot  final" 

Situation:  Aircraft  X  RAY  1  is  turning  onto  the 

final  ILS  approach  at  the  correct 
altitude.  Environmental  conditions  are 
those  selected  for  the  exercise. 

Outgoing  Conmunication:  "X  Ray  1  you  are  cleared  to  land.  Wind 

From  Automated  Controller  now  150  at  20  gusting  27" 

In  the  incoming  signal,  the  automated  voice  controller  system 

recognizes  the  standard  message  content  (approach  control,  -  turning 

to  final)  and  the  unique  content  (X  RAY  1).  The  controller  model  then 
relates  the  standard  message  content  to  the  actual  position  of  the 
aircraft  to  generate  the  response.  In  the  response,  unique  content  (X 
RAY  1  and  wind  data)  are  interlaced  with  standard  content  to  make  up  a 
complete  and  meaningful  message.  For  example,  the  wind  statement  would 
be  different  depending  on  how  the  simulator  environmental  control  is 
set. 

An  automated  ground  controlled  approach  (GCA)  controller  has  been 
successfully  applied  to  an  F-4E  simulation  as  an  integral  part  of  the 
Automatic  Flight  Training  System  (AFTS)  (Swink,  Smith,  Butler,  Futas  and 
Langford,  1975).  This  Instructional  application  of  automated  speech 
technology  marked  the  beginning  of  a  new  era  of  automated  controllers 
for  ATDs.  While  the  F-4E  AFTS  GCA  controller  required  a  resident 
general  purpose  computer  and  a  disk  memory  system  to  provide  a  limited 
repertoire  of  GCA  oriented  words  and  phrases,  modern  microelectronics 
technology  now  offers  similar  capability  on  2  to  4  chips  with 
repertoires  of  up  to  200  words.  One  set  of  chips  converts  analog  speech 
into  compressed  digital  data,  while  the  other  set  works  in  reverse.  The 
speech  conversion  to  drive  the  controller  is  based  on  recognition  of 
acoustic  speech  patterns  and  their  conversion  to  a  small  number  of 
parameters  (normally  between  10  and  20). 

Synthetic  voice-based  controllers  are  expected  to  be  used 
increasingly  for  many  ATD  voice  applications  with  highly  structured 
vocabulary. 

Computer  speech  technology  developments  in  the  last  seven  years  have 
emphasized  "applications"  rather  than  the  development  of  basic 
principles  of  speech  understanding  and  synthesis.  This  has  resulted  In 
certain  system  inadequacies  at  this  time.  For  Instance,  It  is  necessary 
for  each  person  using  a  system  to  Initially  "sign-on"  by  speaking  a 
series  of  standard  or  reference  phrases  to  permit  the  system  to  conduct 
statistical  recognition  of  the  speech  pattern  and  measure  the  duration 


between  words  of  the  speaker  using  the  system  (i.e.  to  "train"  the 
system).  In  an  ATD  environment,  this  initialization  is  somewhat  time 
consuming. 


Automated  speech  understanding  systems  do  not  always  correctly 
recognize  (i.e.  understand)  the  words  and  phrases  spoken  to  them,  even 
though  the  systems  have  been  “trained"  to  recognize  the  words  and 
phrases  involved.  It  is  estimated  that  95  to  99%  correct  recognition 
rates  are  possible  under  ideal  speaking  conditions  (Lea,  1980). 
Recognition  accuracy  was  found  to  range  from  50  to  97%  in  a  recent  field 
evaluation  of  a  prototype  training  system  incorporating  IWR  technology 
(McCauley  and  Semple,  1980).  The  average  correct  recognition  rate  was 
85%.  Thus,  what  presently  is  achievable  is  less  than  what  is  desired. 
However,  the  technology  is  evolving  rapidly,  and  improved  recognition 
accuracy  soon  should  be  achieved.  Even  presently  achievable  recognition 
rates  have  proved  adequate  for  training  and  operational  applications 
where  limited  vocabularies  and  near  ideal  speaking  conditions  exist.  (A 
less  than  ideal  speaking  environment  is  one  that  is  noisy  and/or 
involves  multiple  speakers.) 

Automated  speech  understanding/ computer  synthesized  speech  voice 
controllers  presently  have  significant  limitations.  A  completely 
flexible  system  would  have  to  be  able  to  recognize  thousands  of  words. 
Current  voice  controllers  can  operate  efficiently  using  a  vocabulary 
constrained  by  40  to  60  acoustical  units.  This  appears  to  be  adequate 
for  highly  structured  speech;  however,  it  may  be  many  years  (perhaps  20) 
before  a  person  can  Interface  with  a  computer  through  a  voice  channel 
and  have  the  computer  carry  out  a  series  of  tasks  based  on  unstructured 
conversational  English.  Fortunately,  training  situations  almost  always 
Involve  a  relatively  limited  subset  of  language. 

Related  Support  Features 

Relationships  of  automated  controller  technology  with  other 
Instructional  support  features  are  summarized  below.  Only  the 
relationship  with  automated  performance  measurement  Is  known  to  exist. 
Relationships  with  the  remaining  features  are  speculative  and  deal  with 
possible  future  uses  of  automated  speech  technology. 

Automated  Performance  Measurement.  Automated  controller  systems 
rely  on  automated  measurement.  Automated  measurement  Is  needed  to: 
1)  compare  the  student's  communication  performance  (message  content 
and  timing)  with  what  Is  desired  In  order  to  evaluate  communication 
performance;  and  2)  know  where  the  simulated  aircraft  Is  with 
respect  to  desired  flight  profiles  so  that  appropriate  controller 
messages  can  be  generated  and  transmitted  to  the  student. 

(See  Chapter  IV,  Monitor  and  Evaluate  Performance) 

Automated  Cuing  and  Coaching.  Cuing  Is  the  alerting  of  the  student 
to  take  an  action,  either  as  a  corrective  measure  or  In  order  to 
adhere  to  a  standard  procedure.  Coaching  Is  providing  Instruction 


106 


or  direction.  Computer  speech  technology  offers  one  means  of 
providing  cuing  and  coaching  messages  to  students,  and  checking  for 
the  students  to  acknowledge  receipt  of  the  message.  Automated  cuing 
and  coaching  also  rely  on  automated  measurement.  (See  Chapter  IV, 
Monitor  and  Evaluate  Performance) 

Instructor/Operator  Console  Design.  Using  automated  controller 
technology  in  ATD  designs  could  change  or  even  eliminate  certain 
information  display  requirements  at  the  instructor/operator  console. 
(See  Chapter  VIII,  Instructor/Operator  Console  Design  and  Location) 

Instructor/Operator  Training.  The  use  of  automated  controller 
technology  could  ease  some  instructor/operator  training  requirements 
because  some  of  their  instructional  tasks  could  be  automated.  (See 
Chapter  IX,  Instructor/Operator  Training) 

Instructional  Values 


At  this  time,  assumed  instructional  values  of  automated  controller 
technology  fall  into  the  two  areas  discussed  below.  As  noted 
previously,  the  U.S.  Navy  recently  began  looking  into  the  full  spectrum 
of  training  values  and  applications  for  this  new  and  rapidly  evolving 
technology  (e.g.,  Hicklin,  Barber,  Bollenbacher,  Grady,  Harry,  Meyn  and 
Slemon,  1980;  Clark,  Halley,  Regel son,  Slemon  and  Versteeg,  1979;  Breaux 
and  Goldstein,  1975). 

It  is  assumed  that  automated  controllers  will  unburden  ATD 
instructors  and/or  operators  from  routine,  instructional^  unproductive 
workload.  It  is  expected  that  automating  the  voice  control  of  the 
student  will  allow  the  instructor  to  use  available  time  more 
meaningfully.  Also,  it  is  a  fairly  common  practice  to  require  or  allow 
ATD  operators  to  perform  voice  controller  functions.  Automating 
controller  functions  could  eliminate  the  need  to  provide  ATD  operators 
with  training  on  how  to  function  as  controllers.  Based  on  a  recent 
analysis  (Semple,  et  al.,  1979),  it  is  believed  that  the  following 
controller  tasks  could  be  automated:  approach  and  departure  control; 
ground  control;  tower;  missed  approach  control;  F.A.A.  sector  control; 
carrier  air  traffic  control  center;  marshall  control;  carrier  controlled 
approach,  ground  controlled  approach,  bolter  control;  and  tactical 
intercept  control. 

It  is  assumed  that  automated  controllers  will  markedly  improve 
aircrew  training  and  performance  on  tasks  where  the  student  must  rely 
heavily  on  the  performance  of  various  controllers  in  order  to  be  able  to 
do  his  own  tasks  well,  or  where  the  student  acts  as  a  controller  for 
other  students  during  training.  Automated  controllers,  for  example,  can 
provide  for  consistency  and  correctness  of  air  traffic  control 
communications  in  accordance  with  military  and  F.A.A.  procedures. 
Correct  communication-  procedures  often  are  not  adhered  to  during  ATD 
training.  It  also  was  observed  duri ng • program  site  visits  that  a  number 
of  older  ATDs  simply  do  not  provide  the  instructor  or  operator  with 
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adequate  situation  information  to  use  in  certain  control  tasks,  such  as 
a  GCA  final  approach.  The  retrofit  of  automated  controllers  could 
overcome  this  problem.  Also,  it  was  observed  that  ATD  operators  often 
were  not  adequately  trained  to  function  as  controllers.  Thus,  the 
controller  information  given  to  the  students  often  must  be  questioned  in 
terms  of  its  timeliness  and  accuracy. 

The  following  learning  principles  are  involved  in  properly  designed 
automated  controllers: 

Control  cues  to  ensure  that  the  student  is  forming  the  proper 
associations;  i.e.,  responding  appropriately  to  the  correct  cues. 

Transfer  increases  as  differences  between  reference  (training)  and 
generalization  (job)  stimuli  decreases. 

Ensure  that  the  appropriate  cues  are  available  continually  during 
the  performance  of  a  task. 

Observed  Applications 

No  applications  of  automated  speech  technology  were  observed  during 
site  visits  made  in  this  program.  However,  several  relevant  research 
studies  are  summarized  in  the  following  part  of  this  section  titled: 
Related  Research. 


Utility  Information 

Since  no  applications  of  automated  voice  controllers  were  observed 
during  site  visits,  no  direct  utility  information  is  available. 
However,  the  next  section  of  this  chapter  addresses  graphic  and  text 
readouts  of  controller  information.  Since  these  are  the  present  manual 
alternatives  to  automated  controllers,  the  reader  is  referred  to  this 
section  for  utility  information  on  the  alternatives. 

Related  Research 

Five  training-related  activities  involving  automated  voice 
controllers  technology  were  Identified  during  the  program.  Each  is 
summarized  below  to  provide  the  reader  with  insights  Into  this  evolving 
area. 

The  first  application  of  computer  speech  technology  to  aircrew 
training  was  the  prototype  AFTS  (Swink,  et  al.,  1975).  The  AFTS 
application  Incorporated  computer  speech  understanding  (individual  word 
recognition  technology),  performance  models,  automated  performance 
measurement,  computer  generated  speech,  and  an  early  form  of  automated 
adaptive  training  for  F-4E  CCT  training  In  GCA  and  ground  attack  radar 
(GAR)  weapons  delivery.  Several  Inefficiencies  were  found  during 
prototype  test  in  the  areas  of  performance  modeling,  performance 
measurement,  and  automated  adaptive  training  logics.  However,  computer 


speech  technology  was  found  to  be  sufficiently  acceptable  that  other 
AFTS,  with  some  modifications,  were  purchased  for  use  in  automated 
training  for  A-7  Air  National  Guard  Units  in  GCA  performance.  Other 
AFTS  units  were  purchased  by  the  Greek  Air  Force. 

The  Naval  Training  Equipment  Center  has  sponsored  the  development  of 
an  experimental  prototype  training  device  for  Navy  enlisted  personnel 
precision  approach  radar  controller  training.  This  precision  approach 
radar  training  system  (PARTS)  also  incorporates  individual  word 
recognition  speech  recognition  technology,  PAR  controller  models, 
automated  performance  measurement,  computer  generated  speech,  and 
adaptive  training  logics  (Hicklin,  Nowell  and  Petersen,  1978). 

The  experimental  prototype  PARTS  device  recently  was  evaluated  in  an 
operational  training  environment  (McCauley  and  Semple,  1980).  A 
central  issue  in  the  evaluation  was  to  define  the  capabilities  and 
limitations  of  isolated  word  recognition  technology  for  application  to 
automated  training  of  tasks  that  are  highly  focused  on  the  content  and 
timing  of  voice  messages.  The  evaluation  involved  a  transfer  of 
training  experiment  in  which  the  performance  of  precision  approach  radar 
students  trained  using  PARTS  was  compared  with  the  performance  of 
comparable  students  trained  using  existing,  manually  controlled  training 
devices. 

The  authors  concluded  that  computer  speech  recognition  is 
sufficiently  advanced  to  begin  applying  it  in  appropriate  (highly 
defined  speech-related)  training  tasks.  Isolated  word  recognition  has 
some  limitations,  namely  the  requirements  for  speech  stylization  and 
extensive  speech  sampling  (computer  training).  These  limitations  can  be 
minimized  by  careful  courseware  design,  emphasizing  stylization 
instruction  and  speech  sampling  within  a  task-oriented  instructional 
context. 


McCauley  and  Semple  further  concluded  that  automated  speech 
recognition  provides  the  capability  to:  1)  simulate  a  control  system 
that  includes  verbal  advisories  or  commands,  such  as  air  traffic 
control;  and  2)  expand  the  application  of  computer  assisted  instruction 
by  voice  Interaction  with  the  instructional  system. 

Another  set  of  automated  training  fuctions,  not  necessarily  related 
to  speech  recognition  systems,  includes  adaptive  or  fixed  syllabus 
control,  problem  generation,  record  keeping,  and  performance  feedback  to 
students  and  Instructors.  Combining  the  capabilities  of  automated 
speech  recognition  with  other  automated  training  features  enables  the 
development  of  fully  automated  training  systems  for  speech  related 
tasks.  The  potential  benefits  to  be  derived  from  well  designed 
automated  training  systems  with  speech  recognition  Include  cost  savings, 
the  reduction  of  Instructor  workload,  reduction  of  Instructor  to  student 
ratios,  elimination  of  training  support  personnel  who  are  the 
recipients  of  the  verbal  information.  Increased  student  interest, 
enhanced  training  effectiveness  through  individualized  instruction,  and 
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the  systematic  manipulation  of  a  wide  range  of  task  variables  by  modeling 
and  simulation. 

The  Naval  Training  Equipment  Center  also  has  sponsored  the 
development  of  a  prototype  automated  instructional  support  system  to 
augment  presently  available  instructional  capabilities  of  the  Navy  F-14A 
operational  flight  trainer.  The  development  program  has  many 
instructional  goals.  One  is  the  development  and  test  of  improved 
automated  voice  controller  models  and  related  automated  measurement 
capabilities  for  GCA  and  similar  final  approaches.  The  tests  are  to 
start  in  calendar  year  1980,  with  one  goal  being  the  test,  evaluation 
and  refinement  of  automated  training  technologies  in  an  operational 
setting.  The  work  will  be  performed  at  NAS  Miramar. 

Connected  speech  recognition  system  technology  is  under  development 
by  the  F.A.A.,  the  Navy  and  conmercial  organizations.  New  connected 
speech  recognition  system  technology  is  now  under  evaluation  by  the 
F.A.A.  as  a  means  to  enable  air  traffic  controllers  to  communicate  by 
voice  with  A.T.C.  computer  systems.  The  Navy  has  contracted  to 
establish  ways  of  casting  this  evolving  capability  from  hardware  into 
software,  which  can  be  more  easily  controlled  and  tailored  to  specific 
training  applications.  The  advantages  of  connected  speech  recognition 
technology  are  that  the  speech  understanding  system  component  is  “much 
more  understanding/ forgiving"  of  voice  and  inflection  departures  from 
what  it  originally  heard  during  "computer  training".  In  this  regard, 
readers  of  this  report  are  advised  that  computer  speech  technology 
statements  made  in  this  document  likely  will  have  become  obsolete  during 
the  process  of  printing  the  report.  This  is  an  evolving  technology 
where  further  breakthroughs  will  occur,  leading  to  changes  in  ATD  design 
and  use,  as  well  as  in  weapon  system  design  and  effectiveness. 

Design  Considerations 

The  design  of  automated  controller  models  involves  two  principal 
considerations.  First,  the  technology  of  computer  speech  understanding 
is  improving  very  rapidly.  In  fact,  many  improvements  (e.g.  connected 
speech  understanding)  were  occurring  as  this  report  was  being  written. 
Since  these  and  subsequent  improvements  could  markedly  impact  system 
performance  and  user  acceptance  of  automated  controllers,  system 
designers  should  maintain  a  continuous  awareness  of  the  most  current 
technology  that  is  at  their  disposal. 

A  second  and  important  consideration  is  the  design  of  the 
operational  performance  model  that  drives  the  controller.  Early 
controller  models,  for  example,  were  derived  from  "text  book"  procedures 
for  performing  the  maneuvers  that  were  being  controlled.  In  the 
operational  world,  pilots  seldom  fly  profiles  strictly  according  to 
procedures.  Human  controllers  are  aware  of  this  and  respond 
accordingly.  For  example,  a  pilot  may  choose  to  turn  to  intercept  his 
final  approach  course  at  a  distance  from  touchdown  and  at  an  altitude 
that  an  automated  controller  has  not  been  programmed  to  recognize  as  the 
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initial  point  for  a  final  approach  to  landing.  Two  things  can  result. 
One  is  that  the  controller  model  may  issue  spurious  commands  because  it 
has  not  correctly  recognized  Initial  conditions  for  the  start  of  the 
approach.  For  example.  It  might  command  the  pilot  to  fly  up  to 
intercept  the  desired  glide  path,  rather  than  allowing  him  to  continue 
in  level  flight  until  he  intercepts  the  desired  glidepath.  Second,  the 
automated  performance  measurement  system  that  provides  information  to 
the  controller  model  also  may  be  "fooled"  because  of  a  departure  from 
the  procedure  it  has  been  programed  to  accept  as  baseline  performance. 
This  can  provide  the  controller  model  with  inaccurate  information  and 
can  result  in  a  poorer  score  on  the  approach  than  actually  was  earned. 

The  use  of  controller  models  in  aircrew  training  is  new.  The 
technology  and  the  "lessons  learned"  data  bases  require  expansion.  It 
is  recommended,  therefore,  that  all  automated  controller  models  be 
evaluated  and  refined  during  the  development  process  to  ensure  that  the 
models  function  accurately  and  acceptably  before  their  design  is  frozen. 

GRAPHIC  AND  TEXT  READOUTS  OF  CONTROLLER  INFORMATION 

Summary 

Many  operational  flight  trainers  and  full  mission  simulators  provide 
the  instructor  with  symbolic  situation  awareness  displays  showing  the 
simulated  aircraft's  position  with  respect  to  geographic  reference 
points,  other  aircraft,  or  desired  flight  profiles.  The  types  of 
displays  used  include:  mechanical  plotters;  electronic  displays 
(cathode  ray  tube  displays);  and  readouts.  The  content  of  this  section 
describes  and  discusses  the  advantages  and  disadvantages  of  each  display 
type. 

Feature  Description 

To  date,  instructors  have  been  provided  with  four  types  of 
information  displays  to  use  when  controlling  and/or  monitoring  the 
flight  of  simulated  aircraft.  Each  is  discussed  below. 

Mechanical  plotters  have  been  provided  at  remote  (out  of  cockpit) 
instructor/operator  stations  for  the  past  quarter  of  a  century.  The 
underlying  notion  was  that  Instructors  could  be  provided  with  relative 
aircraft  position  information,  and  that  a  paper  copy  of  flight  paths 
could  be  provided  for  use  during  debriefing.  In  fact,  mechanical 
plotter  displays  are  physically  large;  they  usually  have  been  positioned 
behind  Instructors,  making  their  use  infrequent  and  difficult;  and 
problems  of  obtaining  odd- si  zed  paper  for  them  has  negated  their 
intended  debriefing  use.  Mechanical  plotters  are  now  in  use  (without 
paper  for  debriefing  records)  primarily  to  provide  instructors  with 
pictorial  Information  on  where  the  simulated  aircraft  Is  with  respect  to 
known  geographic  references. 


A  number  of  modern  digital  flight  simulators  incorporate  cathode  ray 
tube  (CRT)  situation  displays  at  instructor/operator  consoles.  These 
displays  have  been  generally  easy  to  interpret.  Content  of  the  displays 
varies  with  the  training  application.  However,  a  typical  display  used 
to  monitor  terminal  area  navigation  performance  and  to  vector  and 
otherwise  control  the  simulated  aircraft  includes:  a  runway;  desired 
glide  slope  and  localizer  paths;  relevant  navigation  aids  such  as  VOR  or 
TACAN  stations;  approach  plate  information;  missed  approach  information; 
natural  hazards,  such  as  mountains  in  the  simulated  area;  and  a  symbol 
showing  present  aircraft  position.  Often,  a  trail  is  produced  behind 
the  aircraft  symbol  to  show  the  last  minute  or  so  of  its  flight  path. 
Similar  displays  could  be  developed  to  show  aircraft  position  in 
relation  to  ground  based  threats. 

A  second  type  of  electronic  display  was  developed  for  use  in 
monitoring  air  intercept  performance  and  for  controlling  a  simulated 
adversary  aircraft  from  a  remote  instructor  console  during  one  versus 
one  air  combat  in  a  Northrop  Corporation  simulator  (Spring,  1976).  The 
central  area  of  a  21  diagonal  inch  graphics  CRT  showed  a  view  of  the 
combat  training  arena  as  it  would  appear  to  an  external  observer  in  a 
third  aircraft.  The  gaming  area  shown  was  20  by  20  miles.  A  grid  of 
lines  was  used  to  represent  the  ground  plane.  The  lines  were  spaced  at 
two  mile  intervals.  Ground  track  lines  were  superimposed  on  the  grid 
under  each  aircraft  symbol  as  an  aid  in  determining  relative  position 
and  movement.  Each  aircraft  was  shown  symbolically  as  a  wedge.  The 
aircraft  controlled  by  the  instructor  had  a  single  vertical  tail;  the 
student's  aircraft  symbol  had  two  tails  to  distinguish  it  from  the 
other.  Also,  the  Instructor's  aircraft  symbol  was  brighter  to  help 
distinguish  it.  Other  highlighting  and  shading  techniques  also  were 
used  to  distinguish  the  two  aircraft  symbols. 

It  was  found  necessary  to  falsely  scale  the  size  of  the  aircraft 
symbols.  After  considerable  experimentation,  a  scale  factor  was 
selected  that  maximized  aircraft  symbol  size  by  reducing  the  symbol  size 
to  only  one  half  as  the  symbol  went  from  a  one  mile  to  a  20  mile  range. 
This  type  of  scaling  was  necessary  so  that  the  aircraft  symbols  did  not 
become  so  small  that  their  orientation  and  direction  of  flight  could  not 
be  perceived. 

A  vertical  column  of  data  along  one  edge  of  the  display  presented 
flight  parameters  for  one  of  the  aircraft;  an  Identical  column  along  the 
other  edge  presented  data  for  the  other.  Parameters  displayed  included 
airspeed,  altitude,  G  loading,  azimuth,  elevation  and  throttle  position. 
A  row  of  data  across  the  top  edge  of  the  display  presented  Information 
on  the  relationship  between  aircraft  (angle  off  tail,  range,  and  closure 
rate).  Pictorial  information  could  be  electronically  "tilted''  from  a 
top-down  view  of  the  engagement  through  a  side  view.  '  A  viewing 
perspective  of  about  45  degrees  was  most  frequently  used  by  Instructors, 
the  Air  Force  Simulator  for  Air  to  Air  Combat  Incorporates  a  similar 
display  at  the  Instructor/operator  console,  as  does  the  Air  Force  Air 
Combat  Maneuvering  Instrumentation  (ACMI)  system. 


A  third  display  type  that  uses  an  array  of  lights  was  observed  on  an 
F-14  OFT  console  for  use  in  carrier  controlled  approaches.  A 
performance  model  determines  whether  the  simulator  was  on  glidepath  and 
on  localizer,  or  not.  For  example,  if  the  simulated  aircraft  was  above 
glidepath,  an  annunciator  light  labeled  “above"  illuminated.  If  the 
student  was  well  above  glidepath,  then  a  second  annunciator  light  labeled 
"well  above"  illuminated.  The  underlying  notions  are  that  the 
instructor/operator  can  read  the  illuminated  message  to  the  student,  and 
that  this  information  is  adequate  to  control  a  GCA  final  approach.  The 
goal  is  to  enable  meaningful  GCA  flight  control  training  without 
burdening  instructors  or  operators  with  a  complex  controller  task.  In 
fact,  this  approach  assumes  a  very  simplified  role  for  the  GCA 
controller.  Also,  trend  information  on  the  progress  of  the  approach  is 
almost  totally  lacking  using  this  display  method. 

A  fourth  display  method  provides  written  text  of  controller  messages 
on  a  CRT  display.  This  method  is  similar  to  the  method  described 
inmediately  above.  The  difference  is  that  the  written  text  approach 
provides  the  complete  text  of  the  appropriate  controller  message  when 
the  message  is  required,  with  the  aim  of  further  standardizing  training 
and  relieving  ATD  instructors  or  operators  of  specific  controller 
knowledge  requirements.  As  with  the  first  method,  the  second  method,  by 
itself,  does  not  provide  adequate  trend  information.  Neither  approach 
can  result  in  a  hard  copy  printout  of  approach  information  for  use 
during  debriefing. 

Both  the  light  array  method  and  the  written  text  method  require 
performance  models  of  desired  GCA  performance  by  the  student,  a 
measurement  system  to  assess  actual  performance  in  relation  to  what  is 
desired,  and  controller  logic  (a  controller  model)  to  drive  the 
displays.  These  are  some  of  the  same  functional  requirements  as  for  an 
automated  controller  system,  and  the  same  design  problems  are  involved. 
(See  the  first  section  of  this  chapter) 

Related  Support  Features 

Relationships  with  three  other  Instructional  support  features  are 
sumnarlzed  below. 


Automated  Performance  Measurement.  Controller  information  displays 
that  tell  tRi  instructor  wFTat  it  Is  (either  generally  or 
specifically)  that  he  is  to  say  as  a  controller  rely  on  automated 
measurement.  Automated  measurement  Is  needed  to  compare  the 
simulated  aircraft's  position  to  desired  flight  profiles  so  that 
appropriate  controller  messages  can  be  generated.  (See  Chapter  V  of 
this  report) 

Instructor/Operator  Console  Design.  The  type  of  controller 
Information  cn splay  that  Is  selected  will  influence  the  physical 
design  of  the  Instructor/operator  console.  Also,  if  an  electronic 


display  medium  is  selected,  the  possible  need  to  timeshare  the 
display  with  other  uses  must  be  considered.  (See  Chapter  VIII  of 
this  report) 

Instructor /Operator  Training.  The  type  of  information  displayed 
will  i nf 1  uence  i nstructor/operator  training  requirements.  The  use 
of  pictorial  (graphic)  information  may  place  additional  requirements 
on  instructor  performance,  and  therefore  training,  because  of  the 
need  to  properly  interpret  and  use  the  available  information. 

Automated  Controller.  The  use  of  automated  controllers  may  relieve 
instructors  from  the  need  to  actively  control  the  situation. 
However,  the  instructor/operator  still  may  desire  a  graphic  display 
of  situational  information  for  trend  data.  (See  the  first  section  of 
this  chapter) 

Instructional  Values 


The  assumed  instructional  value  of  any  graphic  or  text  readouts  of 
controller  information  is  that  they  will  make  possible  training  in  an 
ATD  that  otherwise  would  not  be  possible  if  the  ability  for  an 
instructor/operator  to  act  as  a  controller  was  absent.  Specific 
principles  of  instruction  are  not  involved  in  the  design  of  these* 
displays..  Electronically  generated  graphic  controller  information 
offers  the  added  possibility  of  providing  hardcopy  printouts  of  the 
displayed  information  for  use  as  a  debriefing  aid.  Automated 
controllers  reduce  or  eliminate  the  reed  for  instructor  training  in  the 
controller  function. 

Observed  Applications 

Observed  applications  of  the  use  of  graphic  and  text  readouts  of 
controller  information  during  this  program  are  sumnarized  in  Table  8. 
Examination  of  the  relevant  research  literature  and  results  of  site 
visits  during  the  STRES  program  indicate  that  graphic  displays  and  text 
readouts  of  controller  information  have  been  applied  to  the  following 
program  High  Value  Tasks: 

Landings  and  Takeoffs.  GCA  final  approach  is  the  most  frequent  use 
of  graphic  or  textual  readouts  of  controller  information.  The  Air 
Force  UPT/IFS  simulation  incorporates  a  graphic  display  of  final 
approach  information  and  text  displays.  A  console  operator  acts  as 
the  final  controller.  Similar  graphics  displays  are  incorporated 
into  the  Army's  Device  2B24  and  U.S.  Coast  Guard  ATDs  for  HH-52  and 
H-3  rotary  wing  training.  The  U.S.  Navy’s  F-I4A  operational  flight 
trainer  Incorporates  a  light  matrix  array  of  controller  information 
at  a  remote  Instructor/operator  console. 

Air  to  Air  Combat.  The  SAAC  simulator  incorporates  a  graphic 
presentation  of  air  combat  information.  The  display  is  used  to 
monitor  the  engagement  and  can  be  used  to  control  one  of  the 
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Table  8.  Surveyed  ATDs  Equipped  with  Graphic  or 
Text  Readouts  of  Controller  Information 


Type  of 

Training 

Ai rcraft 

Purpose 

Undergraduate 

Pilot 

Training 

T-37  and  T-38 
(UPT/IFS) 

Vectoring  students 
the  terminal  area, 
GCA  final  approach 

in 

and 

control 

UH-1 

(Device  2B24) 

Vectoring  students 
the  terminal  area, 
GCA  final  approach 

in 

and 

control 

Transition  and 

Continuation 

Training 

HH-52 

H-3 

GCA  final  approach 
control 

adversary  aircraft  from  a  remote  instructor  console.  The  Northrop 
Corporation's  Large  Amplitude  Simulator/Wide  Angle  Visual  System 
(LAS/WAVS)  was  configured  several  years  ago  with  a  display  similar 
to  the  one  found  in  SAAC.  It  was  used  for  the  same  purposes  during 
an  experimental  study  on  the  effectiveness  of  LAS/WAVS  for  training 
basic  one  versus  one  air  combat  maneuvering. 

Utility  Information 

Graphic  (pictorial)  situation  displays  for  approach  control  and 
final  approach  control  were  well  received  by  ATD  instructors  and 
operators  who  had  experience  with  them.  These  displays  provide 
situational  awareness  information  that  Is  not  available  through  text  and 
readout  displays.  The  Instructors/ operators  interviewed  appeared  quite 
proficient  in  using  the  displays  to  perform  their  controller  functions. 
In  one  case  (Air  Force  UPT/IFS),  console  operators  who  functioned  as 
controllers  were  required  to  have  had  prior  air  traffic  control  (ATC) 
experience.  Concern  was  expressed,  in  this  case,  about  the  availability 
in  the  future  of  console  operators  with  this  experience.  Concern  was 
expressed  that  a  considerable  training  investment  would  be  required  to 
achieve  acceptable  controller  performance  from  operators  without  prior 
ATC  experience.  In  a  second  case,  instructors  acted  as  the  controllers. 
They  found  the  graphic  displays  at  their  onboard  consoles  quite  adequate 
for  controlling  a  GCA  final  approach.  However,  it  was  observed  that 
they  did  not  strictly  adhere  to  standard  GCA  procedures  prescribed  by 
the  FAA  and  the  military.  In  a  third  case,  console  operators  were  given 
formal  training  in  the  controller  jobs  they  performed. 

A  text  readout  of  controller  messages  was  observed  on  only  one  ATD 
console.  This  console  also  incorporated  a  graphic  situation  display. 
The  text  readouts  were  not  used  because  all  console  operators  had  prior 
ATC  experience  and  did  not  need  to  refer  to  displayed  text  for  correct 
message  content. 

The  annunciator*  light  array  display  of  controller  information  was 
observed  during  a  different  program  on  a  Navy  F-14  OFT  console. 
Instructors  who  had  used  this  display  method  described  several 
shortcomings.  One  was  a  lack  of  situational  awareness.  A  second  was 
that  the  display  of  GCA  comnands  at  the  console  tended  to  lag  and 
therefore  be  somewhat  untimely.  A  third  problem  was  observed.  This 
method  of  providing  controller  information  is  very  inconspicuous. 
Often,  Instructors/operators  would  be  occupied  with  other  tasks  at  the 
console  and  would  overlook  the  Information  that  was  being  displayed.  As 
a  result,  at  best  the  controller  messages  were  issued  late;  at  worst 
they  were  missed  altogether.  Similar  problems  can  be  anticipated  for 
console  displays  that  present  only  controller  text  messages. 

The  utility  of  air  combat  situation  displays,  like  those  previously 
described,  depends  on  more  than  just  the  single  display,  its  content  and 
organization.  The  Northrop  Corporation  air  combat  situation  display 
described  by  Spring  (1976)  proved  effective  for  monitoring  a  one  versus 
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one  air  combat  engagement  at  a  remote  instructor  console  and  for 
controlling  the  adversary  aircraft  from  the  console.  In  this  case,  the 
console  also  incorporated  an  electronic  display  that  showed  a  horizon 
line,  a  stylized  bowframe  of  the  aircraft  being  controlled,  and  an 
aircraft  symbol  for  the  adversary  aircraft  when  it  was  in  a  front 
quarter  field  of  view.  Also,  at  the  instructor’s  discretion,  the 
display  could  be  switched  to  show  the  same  information  for  the  simulated 
aircraft  being  flown  by  the  student.  In  either  case,  a  forward,  out  of 
the  cockpit  visual  scene  was  simulated.  The  instructor  could  select 
whether  he  wanted  the  scene  as  viewed  from  the  aircraft  he  was 
controlling,  or  from  the  cockpit  from  the  student's  simulator  cockpit. 
Using  the  combat  situation  display  in  conjunction  with  the  other 
electronically  generated  forward  looking  display,  instructors  were  able 
to  control  the  adversary  aircraft  from  the  console  In  very  demanding 
maneuvers,  including  rolling  scissors.  The  point  is  that  aircraft 

situation  displays,  by  themselves,  may  be  Inadequate  for  controlling  a 
highly  maneuverable  threat  aircraft  remotely  from  an  ATD  console. 

The  controls  available  to  the  Instructor  also  were  critical  to  this 
use  of  the  displays  described  by  Spring.  Pitch  and  roll  rate  Inputs 
were  made  through  a  desk  top  mounted  controller.  Controller  inputs  were 
highly  processed  so  that  the  end  result  was  flying  using  control  stick 
steering,  rather  than  conventional  Inputs.  When  the  stick  was  released, 
the  adversary  would  Indefinitely  continue  the  maneuver  established  by 

the  pitch  and  roll  altitude  at  the  time  of  stick  release.  Simple 

throttle  and  speed  brake  controls  also  were  available  to  the 
Instructors.  The  control -display  system  made  it  possible  for 

Instructors  flying  from  the  console  to  . realistically  maneuver  the 
simulated  threat  aircraft.  A  similar  configuration  on  the  McDonnell 
Douglas  MACS  air  combat  simulator  also  Is  reported  to  be  usable. 

The  Air  Force  SAAC  simulator  console  Incorporates  tactical  situation 
display  content  similar  to  that  described  above.  However,  only  one  CRT 
display  is  available,  and  Instructors  must  switch  continuously  back  and 
forth  between  the  tactical  situation  display  content  and  other  display 
content.  The  SAAC  joystick  control  used  to  control  the  adversary 
provides  no  control  feel,  making  it  very  easy  to  grossly  overcontrol  the 
adversary  aircraft  from  the  console.  As  a  result  of  these  combined 
factors.  Instructors  report  that  It  is  very  difficult  to  control  an 
adversary  aircraft  from  the  SAAC  Instructor's  console. 

Apparently  the  SAAC  displays  are  adequate  for  monitoring  progress  of 
one  versus  one  and  two  versus  one  engagements.  SAAC  I/O  console 
features  and  capabilities  are  continuously  being  modified  and  Improved. 
In  one  recent  improvement,  the  "view  from  the  cockpit4  mode 
automatically  switches  to  a  straight  ahead  view  when  the  adversary  Is  in 
the  forward  area,  and  automatically  switches  to  the  360  degree  tactical 
situation  mode  when  he  is  not. 
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Related  Research 


No  experimental  research  studies  were  found  dealing  directly  with 
the  design  or  the  training  use  of  displays  in  satisfying  the  controller 
function. 

Design  Considerations 

The  design  of  controller  information  displays  hinges  on  what  the 
displays  are  to  be  used  for  and  their  Intended  method  of  use.  For 
example,  a  CRT  display  that  would  allow  the  control  of  a  simulated 
flight  enroute  or  In  the  terminal  area  probably  should  present  the 
following  information,  depending  on  the  specific  application:  a  plan 
(downward  looking)  view  of  the  area  being  flown  over;  easily 
distinguishable  symbols  for  each  simulated  aircraft  being  controlled; 
altitude,  speed  and  heading  for  each  controlled  aircraft;  the 
communication  frequency  that  the  simulator  is  tuned  to;  the  navigation 
frequency  that  has  been  selected;  airways  corridors  showing  acceptable 
tolerance  limits;  natural  and  man  made  hazards;  relevant  radio 
navigation  aids  and  their  operational  range;  and  the  runway.  A  display 
used  to  control  a  GCA  final  approach,  on  the  other  hand,  must  present 
separate  displays  of  desired  glidepath  and  course,  and  at  least  the 
approach  end  of  the  runway.  Lines  on  either  side  of  the  desired 

glideslope  or  course  show  maximum  tolerable  flight  path  departures  and  also 
appear  to  assist  instructors  in  performing  the  GCA  controller  role. 
Range  markings  also  should  be  displayed. 

An  electronically  generated  line  showing  the  actual  flight  path 
often  is  useful  for  debriefing.  However,  flight  path  traces  can  result 
in  considerable  clutter.  The  Instructor/operator  should  be  provided 
with  three  easily  selected  trace  options:  no  trace;  a  short  (one  to  two 
inch)  trace;  and  the  full  flight  path  trace.  With  these  options,  the 
Instructor  can  tailor  display  content  to  his  needs,  yet  still  call  up 
the  entire  trace  history  at  the  completion  of  the  flight  to  make  a  hard 
copy  of  the  display  content  for  debriefing.  If  a  hard  copy  capability 
is  desired.  It  Is  Important  to  design  the  display  system  with  a  buffer 
memory  to  retain  the  displayed  image  data  while  the  hardcopy  printout 
process  takes  place.  Otherwise,  the  ATD  would  have  to  be  frozen  while 
the  hard  copy  Is  made.  Interviews  during  the  program  revealed  that  hard 
copy  systems  that  require  the  use  of  ATD  freeze  are  not  used,  while 
systems  Incorporating  a  buffer  memory  are  used. 

Display  clutter  is  to  be  avoided.  Larger  displays,  such  as  18  to  21 
diagonal  Inch  CRT  displays  provide  considerable  display  area  for 

?  resenting  controller  Information.  However,  display  content  should  be 
Imlted  to  the  minimum  consistent  with  the  ability  to  control  the 
flight. 

The  need  for  color  Is  another  consideration.  Color  should  be  used 
only  when  It  is  a  necessary  code  to  enable  Instructors  to  distinguish 
among  the  information  symbols  presented.  The  UPT/IFS  controller  display 
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allows  for  the  simultaneous  terminal  area  control  of  four  ATD  flights. 
It  is  a  black  and  white  system,  and  no  problems  were  noted  that  would 
require  color  coding  for  their  solution. 

The  need  for  dedicated  controller  displays  must  be  considered.  It 
is  becoming  common  practice  to  use  CRT  displays  to  present  many 
different  kinds  of  information  at  ATD  consoles.  If  other  information, 
such  as  malfunction  selection,  must  be  accessed  during  the  period  of 
aircraft  control,  then  it  may  be  desirable  to  dedicate  a  separate 
display  for  controller  information  so  that  the  process  of  controlling  is 
not  continuously  interrupted.  As  a  workable  alternative,  several  ATD 
instructor/operator  consoles  were  observed  (e.g.  Army  Device  2B24)  where 
a  small  portion  of  the  CRT  display,  usually  along  the  bottom  edge  or  at 
a  corner,  was  dedicated  to  the  display  of  relevant  information, 
including:  preselected  malfunction  options;  environmental  options  such 
as  ceiling  height,  runway  visible  range,  and  turbulence  level  to  serve 
as  memory  aids;  and  preselected  initial  conditions  for  reinitializing 
the  ATD. 

Because  of  the  differences  in  the  information  that  must  be 
displayed,  it  often  is  necessary  to  use  one  display  format  for  enroute 
or  approach  control,  and  a  different  format  for  GCA.  This  means 
switching  from  one  display  format  to  the  other  as  the  flight  progresses 
and  controller  information  needs  change.  The  simulated  area  covered  by 
each  display  should  "overlap"  so  that  the  position  of  the  simulated 
aircraft  always  is  known  by  the  controller,  even  when  changing  from  one 
display  to  the  next.  One  ATD  instructor/operator  console  observed 
during  the  program  (Air  Force  UPT/IFS)  had  a  "gap"  between  the 
geographic  areas  used  for  approach  control  and  GCA  displays.  The  result 
was  that  when  the  aircraft  symbol  reached  the  limit  of  the  approach 
control  display,  the  GCA  display  format  could  be  called  up,  but  a  period 
of  time  lapsed  until  the  aircraft  symbol  "flew"  into  the  area  covered  by 
the  GCA  display.  This  was  found  to  be  highly  objectionable.  Also, 
there  is  no  technical  reason  for  designing  displays  in  this  manner.  The 
ATDs  surveyed  all  required  operator  action  to  switch  from  one  controller 
display  to  the  next.  Future  designs  should  provide  instructor/operators 
with  the  option  to  select  automatic  changeover  as  a  function  of  aircraft 
position  or  manual  changeover. 

One  ATD  Investigated  during  the  program  Incorporated  a  capability  to 
store  In  the  computer  approximately  100  minutes  of  graphics  display 
information.  At  the  completion  of  the  training  session,  this  data  base 
could  be  accessed  to  recreate  graphics  display  information  for 
debriefing.  Instructors  reported  that  this  capability  was  used 
relatively  frequently.  This  Army  ATD  (Device  2B24)  was  used  for  initial 
entry  rotary  wing  pilot  training. 

Little  experience  Is  available  on  the  value  of  displaying  the  text 
of  controller  voice  messages  on  controller  displays.  The  one  system 
observed  that  was  capable  of  this  was  relatively  new  (UPT/IFS);  also, 
the  console  operators  Intervlewea  all  had  prior  experience  In  air 


traffic  control.  It  appears  reasonable  to  at  least  consider  this  design 
option  if  relatively  inexperienced  personnel  are  to  act  as  controllers. 
The  approach  assumes,  of  course,  that  a  computer  based  model  can  be 
developed  that  accurately  identifies  controller  message  content  and 
timing.  If  this  can  be  assumed,  it  would  be  reasonable  to  take  the  next 
step  and  display  the  content  of  correct  student  responses  to  allow  for 
Improved  student  feedback.  Additionally,  if  accurate  controller 
messages  can  be  provided  for  visual  display,  the  option  of  using 
computer  generated  speech  to  transmit  the  messages  directly  to  the 
student  should  be  considered  (Semple  et  al.,  1979). 

Mechanical  plotters  are  not  recommended  as  a  means  of  presenting 
controller  information.  Electronic  display  technology  is  much  better 
suited  to  this  application. 

The  approach  of  providing  GCA  controller  information  by  means  of  an 
array  of  annunciator  lights  that  simply  indicate  whether  the  student  is 
high,  low,  left,  etc.,  is  not  recommended.  No  trend  information  is 
provided,  and  such  displays  are  Inconspicuous  and  often  ignored  at 
critical  times.  A  far  better  approach  would  be  to  incorporate  an 
automated  GCA  controller.  (See  the  first  section  of  this  chapter) 

The  design  of  situation  displays  for  the  control  from  a  remote 
console  of  an  adversary  aircraft  during  air  combat  maneuvering  or  for 
the  monitoring  of  air  combat  is  an  evolving  art.  For  example, 
modifications  are  continuously  being  made  to  console  displays  of  the 
SAAC  simulator  to  improve  these  capabilities.  Also,  the  design  of  the 
controls  used  to  maneuver  the  simulated  adversary  may  be  as  Important  as 
the  design  of  the  associated  displays.  The  reader  is  referred  to  the 
second  topic  (Feature  Description)  in  this  section  for  design 
recommendations.  All  that  can  be  said  safely  is  that  any  such  design 
must  be  flexible  and  changeable,  and  should  be  subjected  to  hands-on 
evaluation  and  refinement  exercises  before  the  design  is  frozen. 

COMPUTER  CONTROLLED  ADVERSARIES 


Summary 

Computer  controlled  adversaries  frequently  are  referred  to  as  "iron 
pilots".  They  are  computer  models  that  control  the  flight  path  and 
actions  of  an  adversary  aircraft  based  on  the  tactical  relationship  of 
the  agressor  aircraft  and  the  adversary.  Iron  pilots  have  been  used  in 
visually  equipped  air  combat  simulators  (e.g.  the  Air  Force  SAAC; 
Northrop  Corporation's  LAS/WAVS,  and  NASA's  Differential  Maneuvering 
Simulators).  Properly  designed,  they  can  unburden  the  Instructor  from 
controlling  the  adversary  and  provide  realistic  adversary  maneuvering. 
When  combined  with  an  automated  performance  measurement  capability, 
summary  Information  can  be  generated  describing  engagement  final 
outcome,  offensive/defensive  times  for  each  aircraft,  time  In  gun 
envelope,  time  in  missile  envelope  and  similar  performance  information. 


Feature  Description 

The  original  adaptive  maneuvering  logic  was  developed  under  a  NASA 
contract  and  implemented  in  NASA's  air  combat  device,  the  Differential 
Maneuvering  Simulator.  The  capability  of  this  first  iron  pilot  was 

unrealistic  because  it  had  perfect  information  about  the  attacker.  For 
example,  the  computations  did  not  take  into  account  factors  such  as 
restricted  field  of  view  due  to  simulated  aircraft  wings  or  fuselage 

geometry.  Also,  the  computations  were  nearly  instantaneous,  whereas 
human  pilot  decision  making  requires  some  interval  between  visual  input, 
mental  processing,  decision  making  and  response  action. 

A  version  of  the  original  advanced  maneuvering  logic  was  implemented 
in  the  SAAC.  As  expected,  the  original  system  was  found  to  be 
unrealistic  and  was  judged  to  have  little  training  value  (Kelly,  Brown, 
Van  Arsdall  and  Lee,  1979).  Subsequent  developmental  work  has  resulted 
in  a  much  more  realistic  iron  pilot  that  now  is  used  extensively  for 
control  of  the  adversary  aircraft. 

A  less  sophisticated  iron  pilot  is  described  by  Spring  (1976).  The 
system  described  by  Spring  was  developed  for  use  in  a  transfer  of 

training  experiments  involving  training  on  one  versus  one  basic  air 

combat  maneuvering  in  the  Northrop  Corporation's  Large  Amplitude 
Simulator/Wide  Angle  Visual  System  (LAS/WAVS).  The  purpose  of  the  iron 
pilot  was  to  unburden  instructors  from  having  to  continuously  control 
the  adversary  aircraft,  which  was  done  from  a  remote  console,  and  to 
provide  repeatability  of  adversary  aircraft  actions  during  training.  A 
library  of  canned  adversary  flights  was  tried  initially  as  a  means  of 
meeting  these  goals.  This  approach  was  abandoned,  however,  because 
adversary  flight  *  profiles,  being  canned,  remained  unchanged  regardless 
of  student  pilot  actions.  Instructor  pilots  involved  in  the  program 
felt  that  the  canned  flight  profiles  would  be  of  little  if  any  training 
value.  (The  validity  of  this  instructor  attitude  was  not  tested 
experimentally.) 

The  iron  pilot  developed  for  use  in  the  LAS/WAVS  involved  simulated 
autopilot  loops  around  the  F-40  mathematical  model  used  for  direct 
control  of  the  simulated  adversary.  Autopilot  adversary  control  was 
developed  for  three  maneuvers:  head-on  engagement,  high  yo  yo,  and 
barrel  roll  attack.  The  autopilot  maneuvering  characteristics  were 
programmed  from  instructor  descriptions  of  how  the  adversary  should 
respond  to  attacker  maneuvering.  The  desired  autopilot  maneuvering 
capability  was  selected  by  depressing  one  of  three  push  buttons.  In 
addition.  Instructors  at  the  console  also  could  select  a  “normal" 
adversary  reaction,  or  a  harder  turning  version  of  each  autopilot 
adversary. 

Two  additional  adversary  control  inodes  were  provided.  One  was  a 
control  stick  steering  mode  that  gave  instructors  semiautomatic  control 
of  the  adversary  by  allowing  pitch  rate,  roll  rate  and  throttle  inputs. 
When  the  stick  was  released,  the  adversary  would  indefinitely  continue 
the  maneuver  established  by  the  pitch  and  roll  attitudes  at  the  time  of 
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stick  release.  For  technical  reasons  unique  to  LAS/WAVS,  the  control 
stick  steering  mode  was  limited  to  maneuvers  that  did  not  exceed  85 
degrees  of  bank  or  60  degrees  of  pitch.  The  capability  allowed 
instructors  to  establish  adversary  turns  and  make  adjustments  to  the 
turns  without  having  to  continuously  control  the  adversary.  The 
remaining  mode  was  direct  manual  control,  which  required  continuous 
hands-on  flying.  It  was  possible  to  switch  among  the  three  modes 
(autopilot,  control  stick  steering,  and  manual  control)  during  an 
engagement.  This  capability  was  used  extensively. 

Related  Support  Features 

Computer  controlled  adversaries  are  related  to  the  following 
instructional  support  feature: 

Graphic  Displays  of  Controller  Information.  These  displays 
provide  the  capability  to  monitor  an  automatically  controlled 
engagement,  and  provide  information  for  semiautomatic  or  manual 
control  during  a  simulated  engagement.  (See  the  preceding 
section  of  this  chapter) 

Instructional  Values 


Three  primary  values  are  associated  with  computer  controlled 
adversaries.  One  is  the  repeatability  of  the  behavior  of  the  adversary. 
Adversary  actions  are  consistent  and  predictable  (by  instructors)  during 
training.  This  provides  a  more  consistent  baseline  against  which  to 
evaluate  student  performance  and  diagnose  learning  problems.  The  second 
value  is  the  unburdening  of  the  instructor  during  training  so  that  he 
can  concentrate  on  student  performance,  and  provide  more  meaningful  and 
timely  guidance  and  feedback.  The  third  value  is  the  lessening  of 
specialized  instructor  skills  that  must  be  developed  to  continuously 
control  a  simulated  adversary  aircraft  from  a  remote  console. 

Computer  controlled  adversaries  that  operate  with  selectable  levels 
of  simulated  pilot  skill  hold  considerable  potential  for  air  combat 
training.  Easier  adversaries  could  be  used  earlier  in  training  when  the 
student  is  learning  the  fundamentals  of  air  combat  maneuvering.  As  the 
student's  skill  levels  Increase,  more  difficult  adversary  reactions 
could  be  selected.  In  this  manner,  the  challenge  provided  to  the 
student  would  be  in  keeping  with  his  abilities  to  perform,  rather  than 
overwhelming  him.  The  progressive  approach  to  adversary  capabilities 
could  hasten  the  learning  process.  This  was  the  rationale  behind  the 
"normal"  and  "difficult"  autopilot  adversaries  developed  for  the 
Northrop  LAS/WAVS  simulation.  Instructors  In  LAS/WAVS  used  both 
difficulty  levels  for  the  training  of  transitioning  students. 

Observed  Applications 


The  use  of  an  Iron  pilot  capability  In  the  Air  Force  SAAC  device  was 
discussed  with  Instructors  during  this  program.  Team  members 
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participating  in  the  STRES  program  also  were  involved  in  the  experiment 
to  evaluate  the  effectiveness  of  basic  air  combat  maneuvering  training 
in  the  Northrop  LAS/WAVS  device,  which  incorporated  autopilot 
adversaries. 

Utility  Information 

The  original  iron  pilots,  as  discussed  previously,  were  "too  good". 
They  were  unrealistic  because  they  operated  on  perfect  information  that 
is  not  available  to  the  human  pilot,  and  their  decisions  were  made 
almost  instantly.  This  made  the  first  iron  pilots  virtually  unbeatable. 
They  had  little  training  value  for  these  reasons.  When  iron  pilots  are 
designed  to  provide  realistic  maneuvering,  they  are  well  received  by 
instructors  and  are  used  extensively.  This  was  the  case  in  the  Northrop 
Corporation  study  discussed  previously.  Interviews  with  SAAC 
instructors  also  indicated  that  the  "detuned"  and  more  realistic  iron 
pilot  now  programmed  in  the  SAAC  device  is  used  extensively  and  is 
viewed  by  many  SAAC  instructors  as  being  one  of  the  best  instructional 
support  features  of  the  device. 

Related  Research 


No  research  was  identified  that  investigated  the  design  of  iron 
pilots  for  training  or  the  effects  on  student  performance  of  training 
involving  iron  pilots  versus  human  controlled  adversaries.  Work  to  date 
on  the  development  and  refinement  of  iron  pilots  as  training  aids  has 
been  done  on  the  basis  of  expert  judgement  rather  than  empirical 
research  or  verification. 

Design  Considerations 

Three  design  recommendations  are  warranted  based  on  experience 
gained  with  iron  pilots  in  the  SAAC  device  and  the  Northrop  LAS/WAVS 
device.  The  first  is  that  adversary  actions  controlled  by  the  iron 
pilot  models  must  be  realistic.  The  second  is  that  iron  pilots  with 
selectable  "skill  levels"  should  be  developed  so  that  adversary 
responses  can  be  adjusted  to  student  pilot  skill  levels  during  training. 
The  third  recommendation  is  to  incorporate  very  fundamental  or  basic 
adversary  maneuvering  capabilities,  such  as  simple  turning  maneuvers,  to 
be  used  early  in  basic  air  combat  training.  This  capability  was  used 
extensively  in  the  Northrop  Corporation  study  discussed  previously. 
With  respect  to  this,  the  control  stick  steering  concept  used  by 
Northrop  also  was  used  extensively  to  make  adjustments  to  adversary 
turning  responses. 

The  iron  pilot  maneuvering  logic  programmed  in  SAAC  has  one  notable 
deficiency  that  should  be  overcome  In  future  applications.  The  SAAC  Is 
capable  of  simulating  two  versus  one  air  combat  engagements.  The  Iron 
pilot  logic  In  SAAC  is  programed  to  respond  only  to  the  closest 
aircraft.  As  a  result,  for  example,  if  the  iron  pilot  adversary  is 
attacking  one's  wingman,  the  other  pilot  can  simply  fly  close  to  the 
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Iron  pilot  adversary  to  make  It  react  to  him,  allowing  the  wlngman  to 
escape.  This  Is  not  realistic,  and  allows  "simulator  wise"  pilots  to 
defeat  the  intent  of  the  training  being  provided. 


I 
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CHAPTER  VII 

PREPARE,  BRIEF  AND  DEBRIEF 


FUNCTION  DEFINITIONS 

Three  instructional  functions  and  their  related  support  features  are 
addressed  in  this  chapter.  Each  function  is  defined  below. 

Prepare  means  working  out  beforehand  the  details  of  what  is  to  be 
trained  and  how  it  is  to  be  trained  during  a  period  of  ATD  instruction. 
For  the  purposes  of  this  program,  prepare  focuses  on  individualized, 
proficiency  advanced  training  where  the  instructor  is  required  to  make 
training  content  decisions  as  a  matter  of  routine. 

Brief  means  giving  final  training  session  instructions  to  the 
student,  coaching  him  through  the  events  of  the  mission  in  advance, 
questioning  him  to  determine  his  readiness  for  upcoming  training,  and 
providing  remedial  instruction  in  weak  areas.  In  short,  it  is 
cognitive  (mental)  pre-training. 

Debrief  means  to  review  with  the  student  the  training  events  that 
occurred  during  an  ATD  session  with  the  goals  of  evaluating  performance, 
providing  verbal  instruction  to  aid  the  student  in  overcoming 
performance  deficiencies,  and  providing  positive  feedback  for  tasks 
performed  wel 1 . 

RELATEO  FUNCTIONS 

The  reader  is  directed  to  the  chapters  listed  below  that  deal  with 
functions  related  to  preparing,  briefing  and  debriefing.  The  reader 
also  is  directed  to  Chapters  VIII  and  IX  of  the  utilization  report  for 
related  information  on  assessing  and  maintaining  ATD  training 
effectiveness. 

Chapter  IV,  Monitor  and  Evaluate  Performance 

Chapter  V,  Control  and  Individualize  Training 

INSTRUCTIONAL  SUPPORT.  FEATURES  ADDRESSED 

The  instructional  features  addressed  in  this  chapter  are: 

Computer  managed  instruction 

Recorded  briefings 

Automated  performance  measurement 


Instructor  console  station  displays 

Hard  copy  printouts  of  performance  information 


The  balance  of  this  chapter  is  organized. differently  from  preceding 
chapters  due  to  the  differing  nature  of  the  materials  covered.  Separate 
sections  deal  with  each  function  (prepare,  brief  and  debrief). 
Sub-sections  address  relevant  instructional  support  feature  technology. 
Automated  performance  measurement  and  instructor/operator  console  design 
are  discussed  in  detail  in  Chapters  IV  and  VIII  respectively. 

PREPARE  FUNCTION 

Summary 


Computer  managed  instructional  (CMI)  systems  hold  future  promise  for 
assisting  in  preparing  ATD  training  session  content.  CMI  systems 
compare  a  student's  training  history  with  a  standard  training  syllabus 
made  up  of  lists  of  clearly  defined  tasks.  Student  performance  is 
measured,  and  tasks  are  identified  where  acceptable  performance  has  and 
has  not  been  achieved.  Individualized  training  then  is  developed  and 
scheduled.  Improved  operational  readiness,  training  efficiency,  and 
reduced  instructor  administrative  workload  are  assumed  benefits. 
Computer  management  of  ATD  instruction  is  just  coming  into  being. 
Therefore,  practically  based  design  and  use  information  remains  scarce. 

Computer  Managed  Instruction 

CMI  systems  can  be  designed  to  perform  many  functions.  Their 
general  function  is  a  tool  in  the  management  of  instruction  so  that 
training  is  efficiently  directed  toward  desired  outcomes  on  an 
individual  student  basis.  In  a  general  sense,  CMI  systems  compare  a 
student's  training  history  with  a  standard  training  syllabus.  The 
syllabus  consists  of  a  list  of  clearly  defined  tasks  the  student  is 
required  to  perform,  together  with  criteria  of  acceptable  performance. 
The  CMI  system  measures  (either  automatically  or  from  manual  inputs) 
student  performance  on  each  task  performed,  and  identifies  those  tasks 
the  student  can  perform  and  those  he  still  cannot.  System  software  then 
composes  an  individualized  set  of  instructional  tasks  to  be  trained 
during  subsequent  training  sessions.  Sophisticated  systems  can  schedule 
subsequent  training  based  on  training  asset  availability,  and  predict  a 
training  completion  date. 

CMI  systems  can  be  designed  to  control,  or  manage  various  sized 
units  of  instruction.  They  may  range  from  entire  blocks  of  instruction, 
made  up  of  a  number  of  training  tasks  requiring  the  same  or  different 
devices,  to  individual  training  tasks.  A  CMI  capability  that  could  be 
used  to  highly  individualize  training  in  ATDs  would  have  to  deal  with 
the  measurement,  assessment  and  scheduling  of  training  at  a  very  precise 
task  level.  This  is  necessary  so  that  the  tasks  trained  within  a 


particular  block  of  time  could  be  tailored  precisely  to  student  learning 
needs. 

Related  Support  Features 


Computer  management  of  ATO  training  is  related  to  the  following 
instructional  support  features  addressed  in  this  report: 

Automated  Performance  Measurement.  An  automated  performance 
measurement  capability  is  one  means  that  can  be  used  to  inform  the 
CM1  system  of  the  acceptability  of  student  performance  on  a  task  by 
task  basis.  (See  Chapter  IV)  Alternately,  a  comparable  instructor 
entry  is  required. 

Automated  Adaptive  Training.  Automated  adaptive  training  is 
computer  controlled  training  in  which  the  problem,  cues  or  tasks  are 
varied  as  a  function  of  how  well  the  student  performs.  Such  systems 
determine  which  tasks  the  student  has  and  has  not  yet  mastered,  and 
can  make  detailed  changes  to  training  content  within  a  training 
session.  (See  Chapter  V) 

Instructional  Values 

Four  instructional  values  often  are  discussed  for  computer 
management  of  aircrew  ATD  training.  The  first  is  CMI's  support  of 
operational  readiness  by  ensuring  that  all  tasks  are  trained  to 
accepted  and  standardized  criteria  of  performance.  A  second  value  is 
improved  training  efficiency,  because  ATD  training  is  focused  on 
overcoming  performance  deficiencies,  rather  than  less  productive  lock 
step  training  where  training  time  continues  to  be  spent  on  tasks  which 
already  have  been  mastered.  Third,  many  assume  it  to  be  beneficial  that 
students  can  progress  through  training  at  their  own  pace.  Thus,  both 
fast  and  slow  learners  can  be  accommodated  while  ensuring  that  all 
master  the  required  tasks.  It  must  be  recognized,  however,  that 
self-paced  proficiency  advancement  through  training  programs  also  can 
mean  that  the  pipeline  must  be  able  to  accept  newly  trained  or 
transitioned  pilots  whenever . they  complete  training.  Administratively, 
this  can  pose  problems.  One  solution  to  this  problem  used  in  other 
training  contexts  has  been  to  provide  those  who  finish  minimum  couse 
requirements  early  with  additional,  "enrichment"  training  on  tasks  that 
are  job  related  but  are  not  part  of  the  training  syllabus  or  typically 
are  not  trained  to  high  levels  of  proficiency  (McCauley  and  Semple, 
1980).  This  is  a  productive  use  of  training  time,  and  is  a  method  for 
controlling  graduation  dates  while  taking  advantage  of  self-paced, 
proficiency  advanced  training.  Finally,  CMI  systems  can  relieve 
Instructors  of  time  consuming  administrative  details  involved  with 
scoring  and  tracking  student  progress  and  proficiency  on  a  task  by  task 
basis. 


The  following  learning  principles  are  involved  in  properly  designed 
and  used  ATD  computer  managed  instructional  systems: 

Ensure  that  relevant  subordinate  capabilities  have  been  thoroughly 
learned  before  counting  on  them  to  support  the  learning  of  more 
advanced  capabilities. 

Objectively  measure  the  frequency  and  types  of  errors 
throughout  training. 

Even  a  partial  CM1  capability  may  benefit  training  effectiveness  and 
efficiency.  Semple,  Vreuls,  Cotton,  Durfee,  Hooks  and  Butler  (1979) 
have  described  a  prototype  instructional  support  system  that  tracks 
student  progress  in  achieving  acceptable  performance  for  a  number  of 
aircrew  tasks.  The  system,  presently  being  constructed,  does  not  use 
this  information  to  create  subsequent  training  exercises.  However,  the 
design  calls  for  a  capability  to  provide  instructors  and  students  with  a 
listing  of  tasks  that  have  been  mastered,  and  those  yet  requiring 
mastery.  It  is  expected  that  providing  even  this  level  of  information 
will  enable  more  efficient  tailoring  of  instruction  to  student  needs, 
although  the  tailoring  would  be  done  "manually". 

Observed  Applications 

No  computer  managed  instructional  systems  designed  specifically  for 
aircrew  training  were  observed  during  program  site  visits.  However,  a 
number  of  observations  were  made  during  this  and  prior  programs  that 
bear  on  future  designs  and  uses  of  these  systems.  The  observations  are 
summarized  below. 

Instruction  in  ATDs  usually  is  designed  to  be  administered  in 
blocks.  The  length  of  blocks  vary,  but  usually  range  from  one  to  four 
hours  in  duration.  The  length  of  the  block  often  is  selected  for  ease 
of  administration.  Training  to  be  addressed  in  each  block  of 
instruction  Is  defined,  but  the  level  of  definition  often  is  marginally 
adequate  at  best.  On  the  other  extreme,  one  site  was  visited  where 
highly  detailed  training  objectives  were  specified  for  each  one-hour 
block  of  instruction.  The  objectives  consumed  15  to  20  pages  of  written 
description.  Although  this  level  of  detail  may  be  desirable  for 
Instructor  training,  it  was  proving  unworkable  in  day  to  day  operations. 

It  has  been  observed  in  a  number  of  cases  that  specific-  training 
tasks  or  objectives  intended  for  a  block  of  instruction  often  are  not 
attempted  due  to  training  time  running  out.  Training  thus  missed  may  or 
may  not  be  addressed  in  subsequent  blocks  of  instruction.  Frequently, 
record  keeping  is  quite  informal,  and  on  occasion  it  was  observed  that 
the  student  Is  the  one  tasked  with  keeping  track  of  which  training  he 
has  and  has  not  received.  Consequently,  in  some  cases,  there  is  no 
assurance  that  tasks  Intended  to  be  trained  in  the  ATD  even  were 
attempted. 
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A  properly  designed  computer  managed  instructional  system  can  be 
used  to  keep  track  of  which  tasks  have  been  attempted  by  the  student, 
and  which  of  these  he  has  and  has  not  mastered.  This  information  then 
can  be  used,  together  with  appropriate  computer  decision  logics,  to 
tailor  content  of  the  next  block  of  instruction  to  individual  student 
needs.  As  discussed  previously,  some  systems  also  can  be  designed  to 
schedule  subsequent  training  for  each  student  in  keeping  with  his  needs 
and  the  availability  of  instructional  assets.  These  capabilities  of 
computer  managed  instructional  systems  can  assist  in  easing  the 
administration  of  training  while  also  tailoring  training  to  individual 
needs,  which  might  be  cumbersome  if  done  manually.  (See  Chapter  V  of 
the  utilization  report  for  addition  information  on  sequencing  and 
scheduling  ATD  training.) 

Utility  Information 

Utility  information  is  not  available  because  no  applications  of  CMI 
were  observed  during  the  program  that  dealt  specifically  with  aircrew 
ATD  training. 

Related  Research 


No  experimental  research,  as  such,  was  identified  that  dealt 
specifically  with  the  evaluation  of  alternative  designs  or  uses  of  CMI 
for  ATD  training.  Two  computer  information  management  systems  are  being 
developed  and  implemented,  however,  that  are  related.  The  Air  Force 
Operations  Resource  Management  System  (AFORMS)  is  being  developed  to 
provide  conmanders,  unit  schedulers  and  training  officers  with  timely 
information  for  critical  resource  allocation  decisions.  AFORMS  performs 
a  number  of  functions.  One  will  be  to  keep  track  of  and  provide 
information  for  scheduling  the  events  to  be  trained  during  flight 
sorties.  In  this  respect,  AFORMS  shares  a  commonality  with  ATD  CMI 
systems  in  that  master  listings  of  unit  level  flight  training 
requirements  must  be  maintained,  together  with  current  information  on 
the  meeting  of  each  requirement  by  individual  aircrew  members. 
Requirements  to  be  met  during  subsequent  flights  then  can  be  determined 
and  scheduled  on  an  individual  aircrew  member  basis. 

The  U.S.  Navy  is  implementing  an  Aviation  Training  Support  System 
(ATSS).  The  overall  objectives  of  ATSS  are  directed  toward  maintaining 
personnel  readiness  in  the  face  of  economic  restrictions,  manpower 
competition,  and  increasing  weapon  system  sophistication.  ATSS  involves 
a  variety  of  training  applications,  including  fleet  readiness  training. 
A  CMI  subsystem  directs  student  learning  activities  and  manages  and 
tracks  the  performance  and  progress  of  students  through  self-paced, 
individualized  training  exercises  (Naval  Weapons  Center,  1978). 

Design  Recommendations 


Specific  design  recommendations  are  difficult  to  make  for  ATD 
computer  managed  instructional  systems.  For  one,  these  usually  are  very 


complicated  systems.  Second,  no  actual  experience  has  been  gained  to 
date  using  CMI  systems  for  aircrew  training.  Therefore,  the  following 
recommendations  are  general.  They  are  adapted  from  Semple,  et  al. 
(1979). 

The  master  task  listing  against  which  student  progress  is  measured 
must  be  specific.  This  is  no  small  job.  For  example,  specific  normal 
and  emergency  procedures  that  are  to  be  trained  must  be  separately 
identified  so  that  tracking  and  planning  of  training  can  be  done 
precisely.  Similarly,  maneuvers  to  be  trained  must  be  specifically 
defined.  Semple,  et  al.,  for  example,  identified  nine  different 
approaches  to  shore  based  facilities  for  a  Navy  program  they  analyzed. 
These  included:  Hi-TACAN  A,  Miramar;  Random  Radar  Vector  to  Hi-TACAN  A 
initial  approach  fix;  and  random  radar  vector  to  GCA  pickup,  MCAS,  Yuma. 
Other  flight  tasks  were  similarly  defined  so  that  student  performance  on 
each  could  be  readily  determined.  Precision  is  necessary  to 
individualize  draining  in  ways  that  promote  efficient  use  of  training 
time  and  other  assets. 

A  second  general  requirement  appears  to  be  flexibility  of  use.  It 
takes  fairly  sophisticated  computer  logic  to  review  a  list  of  tasks  yet 
to  be  trained,  and  construct  a  meaningful  ATD  training  exercise  from  the 
list.  The  U.S.  Navy  is  sponsoring  work  to  develop  such  logics  for  the 
detailed  level  of  planning  discussed  above.  Until  such  logics  are 
developed  and  verified,  it  is  necessary  to  rely  on  instructor 
participation  in  individualizing  training  session  content.  This  can  be 
accomplished  by  making  available  to  him  the  listings  of  tasks  trained 
and  yet  to  be  trained  that  are  available  within  CMI  systems. 

Finally,  task  lists  and  computer  logics  that  use  them  must  be 
relatively  easy  to  modify.  This  is  necessary  because  training 
requirements  frequently  change,  and  computer  systems  supporting  training 
must  be  correspondingly  kept  up  to  date. 

A  final  issue  is  the  size  of  the  training  program  needed  for  CMI  to 
be'  economically  advantageous.  Smaller  programs  may  not  be  worth  the 
computing  and  software  investments.  In  such  cases,  "manual"  tailoring 
of  instruction  may  be  more  practical.  The  minimum  program  size  issue 
was  not  addressed  during  STRES. 

BRIEFING  FUNCTION 

Recorded  Briefings 

Much  aircrew  training  in  ATDs  now  is  provided  in  organized  blocks  of 
Instruction.  Commonly,  instructors  brief  students  before  each  ATD 
session.  Many  times,  the  only  preparation  the  student  goes  through  for 
the  ATD  session  Is  the  pre- session  briefing.  Often,  tasks  to  be 
trained,  performance  standards  expected,  and  the  sequence  in  which 
events  will  occur  are  not  clearly  understood  by  the  student  when  he 
enters  the  ATD*  Other  times,  students  use  ATDs  for  self  instruction. 
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and  guidance  on  what  they  should  do  during  self-instruction  sessions 
would  be  beneficial.  Recorded  briefings,  with  accompanying,  supporting 
text,  hold  promise  for  improving  the  efficiency  of  ATD  training 
sessions,  whether  or  not  an  instructor  is  present,  because  task 
description,  coaching  and  related  study  information  could  be  thoroughly 
studied  by  students  prior  to  their  training  in  ATDs.  Automated 
briefings  also  can  play  significant  roles  during  automated 
demonstrations  (See  Chapter  III)  by  providing  tailored  instruction  and 
commentary  that  is  in  direct  keeping  with  the  demonstration. 

Discussion 


Recorded  briefings  are  succinct,  well  organized,  clearly  stated, 
recorded  messages  dealing  with  the  content  and  performance  expectations 
of  structured  ATD  training  sessions.  They  may  be  used  by  instructors  in 
lieu  of  portions  of  individual  briefings,  or  they  may  be  used  by 
students  for  more  systematic  study  of  task  requirements  to  be  addressed 
during  the  ATD  training  session.  Supplemental,  but  brief  text  may  also 
be  desirable  so  students  can  review  details  of  tasks  and  performance 
expectations  confronting  them.  Recorded  pre-session  briefing,  together 
with  accompanying  text  dealing  with  performance  details,  offer  good 
potential  for  improved  standardization  and  efficiency  of  highly 
structured  ATD  training. 

Recorded  pre- session  briefings  were  used  at  two  training  sites 
visited.  They  were  used  in  conjunction  with  automated  demonstrations  in 
an  Army  initial  entry  rotary  wing  training  program.  It  was  not  possible 
to  determine  with  any  accuracy  how  often  the  recorded  briefings  were 
used.  Some  instructors  used  them  fairly  regularly;  others  hardly  at 
all.  Their  use  was  left  to  the  instructor's  discretion,  and  general  use 
guidelines  were  not  issued.  It  was  pointed  out,  however,  that 
experienced  pilots  using  the  simulators  to  practice  instrument  flight 
before  check  rides  also  used  automated  instrument  flight  demonstrations 
and  accompanying  recorded  briefings.  One  commercial  airline  also 
provided  recorded  briefings  for  in-simulator  use  by  flight  engineers  who 
were  using  the  simulators  during  off-hours  to  practice  instrument  flight 
skills.  In  this  application,  qualified  instructors  often  were  not 
available.  The  recorded  briefings  dealt  with  simulator  operation, 
aircraft  performance  data,  and  flight  control  techniques. 

Since  recorded  briefings  assume  that  the  training  content  of  the  ATD 
session  is  defined,  this  feature  is  related  to  programmed  mission 
scenarios  and  automated  demonstrations  (See  Chapter  III  of  this  report). 
Training  content  does  not  need  to  be  under  computer  control,  however,  as 
would  be  the  case  with  automated  demonstrations  or  programmed  scenarios. 
Rather,  a  device  operator  or  Instructor  could  work  from  a  printed 
mission  scenario  that  details  task  sequences  and  requirements  making  up 
the  training  session. 
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Many  hours  often  are  spent  by  instructors  and  students  reviewing 
training  events,  contingency  plans  and  safety  considerations  before 
training  flights  are  undertaken.  The  method  being  used  is  cognitive 
(mental)  pre-training.  The  method  is  essential  not  only  for'  safety  of 
flight,  but  also  for  productive  use  of  training  resources.  The  use  of 
recorded  ATD  session  briefings  and  related  sunmary  text  information  also 
involves  cognitive  pre-training.  Actual  hands-on  experience,  then, 
becomes  more  practice  oriented  than  discovery  oriented.  This  is  the 
primary  assumed  instructional  value  of  recorded  briefings. 

DEBRIEFING 

The  instructional  features  addressed  in  this  section  are  dealt  with 
in  detail  elsewhere  in  this  volune.  Therefore,  this  chapter  has  been 
organized  around  known  and  possible  roles  that  each  feature  can  play  in 
support  of  debriefing.  Crossreferencing  is  used  to  direct  the  reader  to 
detailed  information  on  the  individual  features  addressed. 

Role  Of  Automated  Performance  Measurement 

Summary.  A  properly  designed  automated  performance  measurement 
(APM)  system  should  be  a  valuable  debriefing  aid  because  of  the 
objectivity  and  completeness  of  the  performance  information  that  can  be 
made  available  to  the  student  and  instructor.  Unfortunately,  most 
existing  APM  systems  are  so  poorly  designed  as  to  be  useless.  The 
amount  of  detail  they  present  is  overwhelming,  and  the  performance 
details  often  cannot  be  related  to  specific  mission  events.  Automated 
measurement  and  scoring  of  discrete  event  procedures  (i.e.  switch 
positioning  procedures)  is  on  the  horizon,  but  not  yet  available 
operationally.  However,  a  number  of  new  ATDs  will  display  procedural 
event  sequences  on  CRT  displays  for  instructor  review  and  evaluation 
(e.g.,  C-130,  B-52,  F-16  and  A-10).  It  is  believed  by  experts  in  the 
field  that  a  useful  APM  system  would  provide  summary  score  (grade-like) 
outputs  separately  for  each  task  trained.  Additional  performance 
details  could  be  obtained  from  the  system  at  the  instructor  or  student's 
discretion.  Prototypes  of  such  APM  systems  are  being  developed  for 
field  test.  (See  Chapter  IV  of  this  report) 

Discussion.  Presently,  instructors  make  notes  during  training  to 
serve  as  memory  aids  during  debriefing  and  for  completing  grade  forms. 
A  well  designed  APM  system  could  complement  or  replace  the  need  to  take 
notes.  Grade  forms  that  typically  are  used  identify  training  tasks  in 
rather  global  ways.  For  example,  grades  often  are  required  for 
"approach  and  landing",  but  specifics  of  the  particular  type  of  approach 
and  final  approach  flown  (e.g.  TACAN  approach  to  an  initial  approach 
fix,  followed  by  a  GCA  final  approach  with  no  flaps)  are  not  called  out. 
An  APM  system  needs  to  know  such  specifics  so  that  it  can  measure  and 
score  appropriate  performance  dimensions.  Details  of  tasks  trained 
could  easily  be  made  available  to  instructors. 


Physical  design  and  location  of  the  APM  display  will  affect  its 
training  value  and  use  during  debriefing.  If  performance  information  is 
displayed  electronically  at  an  instructor's  console  CRT  display, 
problems  may  arise  because  the  console  cannot  be  tied  up  for  debriefing 
due  to  the  demands  of  a  subsequent  training  session  that  is  about  to 
start.  Hard  copy  printouts  can  be  used  to  overcome  this  problem,  if  the 
printout  process  also  does  not  tie  up  the  console  or  the  display. 

One  instance  was  observed  during  the  program  where  relatively 
simple,  but  meaningful  automated  measurement  information  was  output  on  a 
line  printer.  However,  the  line  printer  was  nowhere  near  the 
instructor/ operator  console;  in  fact,  it  was  in  the  ATD  computer 
complex.  Needless  to  say,  it  was  not  used  for  training. 

Role  Of  Computer  Managed  Instruction 

Summary.  Computer  managed  instructional  (CMI)  systems  can  be 
designed  to  perform  many  instructional  management  functions.  One 
function  is  keeping  track  of  which  training  objectives  have  and  have  not 
been  met  by  each  student  and  aiding  in  determining  the  instructional 
content  of  the  next  ATD  training  session.  One  goal  of  CMI  systems  is  to 
individualize  training  in  order  to  make  the  most  efficient  use  of 
training  assets.  CMI  system  output  can  be  used  during  debriefing  to 
identify  for  the  student  the  specific  training  objectives  he  achieved 
during  the  session  and  to  be  addressed  in  his  next  ATD  session.  This 
Information  should  be  useful  in  assisting  him  to  prepare  for  the  next 
session.  (See  the  section  of  this  chapter  addressing  the  prepare 
function.) 

Discussion.  CMI  systems  have  not  yet  been  applied  to  aircrew  ATD 
training.  It  appears,  however,  that  they  soon  may  be.  One  of  the 
assumed  advantages  of  CMI  is  that  it  will  enable  tailoring  of  ATD 
training  to  student  learning  needs  during  a  particular  session.  Thus, 
it  may  not  be  possible  for  students  or  instructors  to  refer  to  printed 
syllabi  and  other  course  description  documents  to  Identify  the  content 
of  the  next  ATD  session.  It  would  appear  reasonable,  therefore,  to 
interrograte  the  CMI  system  during  debriefing  to  identify  the 
instructional  content  of  the  next  ATD  session  so  that  the  student  can 
prepare  properly. 

Role  Of  Console  Displays 

Summary.  A  number  of  modern  military  ATDs  Incorporate  graphic 
displays  of  flight  profile  and  related  information  at  Instructor 
consoles.  Some  have  capabilities  for  making  hard  copy  printouts  of 
display  content.  Where  such  capabilities  exist  they  have  been  used 
during  debriefing,  particularly  for  undergraduate  and  transition 
training. 
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Discussion.  Three  ATD  training  systems  observed  during  program  site 
visits  provided  the  capability  of  replaying,  in  real  time,  flight 
profile  performance  in  relation  to  desired  courses.  Two  were  rotary 
wing  training  devices;  the  third  was  a  UPT/IFS  complex.  These  systems 
provided  a  downward  view  where  flightpath  history  was  displayed  in 
relation  to  ground  references.  One  other  system  displayed  history  of 
air  combat  profiles  from  several  different  perspectives.  A  similar 
system  was  used  in  a  research  study  of  air  combat  maneuvering  training. 
Several  new  ATDs  will  incorporate  this  feature  (e.g.  B-52  and  C-130). 
In  most  cases,  instructors  comnented  that  graphic  displays  of  flight 
profile  histories  were  useful  during  debriefing  because  they  provided 
detailed,  easy  to  understand  memory  aid  information.  A  problem  that 
frequently  made  it  impossible  to  use  such  displays  was  the  press  of 
time,  where  the  console  was  needed  for  other  training  uses. 

One  potential  solution  to  the  time  problem  is  to  provide  for  remote, 
off-line  replay  capabilities.  Only  one  prototype  system  was  identified 
that  had  a  remote  replay  capability.  The  remote  replay  reportedly  was 
not  used,  but  no  records  were  kept,  and  it  is  totally  unclear  why  this 
was  the  case.  The  U.S.  Navy  is  developing  an  automated  instructional 
support  system  that  will  include  remote  replay.  The  system  will  not  be 
ready  for  operational  try  out  until  late  1980,  however,  so  potential 
benefits  remain  speculative.  The  Air  Force  Simulator  for  Air-to-Air 
Combat  (SAAC)  incorporated  a  TV  taping  system  to  record  graphic 
flight  profile  data  from  an  instructor  console  display.  Technical 
difficulties  precluded  its  use.  However,  pilots  seem  to  be  "picture 
oriented"  people.  Further  work  is  required  to  determine  whether  graphic 
flight  profile  information  in  fact  benefits  aircrew  member  training;  the 
assumption  obviously  is  that  it  will. 

Role  Of  Hard  Copy  Printouts 

Summary.  Hard  copy  printouts  of  performance  information  are  used  by 
instructors  for  debriefing  where  the  capability  exists,  and  where 
printouts  are  designed  for  convenient  access. 

Discussion.  As  discussed  initially  in  this  section,  printouts  of 
detailed,  numerical  automated  measurement  data  are  not  used.  This  is 
because  present  measurement  systems  are  early  generation  capabilities, 
and  provide  little  if  any  usable  data.  One  air  combat  manuvering 
simulator  system  (SAAC)  was  observed  where  simple,  yet  meaningful 
performance  data  were  available  via  printout.  The  data  dealt  with  time 
within  firing  envelopes,  number  of  weapon  strikes,  and  related 
information.  In  this  case,  however,  the  hard  copy  printer  was  so 
physically  remote  from  the  instructor's  console  that  It  was  not  used. 
This  emphasizes  the  need  for  convenience. 

The  ability  to  obtain  a  debriefing  hard  copy  of  graphic  flight 
profile  Information  was  viewed  favorably  by  Instructors.  Where  the 
capability  existed,  it  was  used  by  many.  One  ATD  produced  performance 


134 


alert  symbols  overlaying  flight  profile  data.  This,  too,  was  viewed 
favorably  as  a  debriefing  aid. 

It  is  necessary  to  incorporate  a  display  buffer  capability  for  hard 
copy  printout.  With  a  buffer,  hard  copy  print-outs  can  be  made  while 
the  display  is  used  for  a  different  purpose.  Without  a  buffer,  the 
entire  simulation  may  be  tied  up  while  a  hard  copy  printout  is  made. 
This  virtually  precludes  making  hard  copy  printouts  during  an  ATD 
training  session,  because  neither  instructors  nor  students  want  to  halt 
(freeze)  training  just  to  get  a  hard  copy  performance  printout. 

Discrete  event  (i.e.  switchology)  procedures  measurement 
capabilities  are  just  beginning  to  mature.  It  is  unclear  at  this  time 
just  how  much  procedural  detail  will  be  useful  during  debriefing.  It  is 
possible  to  inundate  students  and  instructors  with  hard  copy  of 
procedural  detail.  This  approach  surely  will  make  such  information 
unusable.  How  to  best  design  procedural  hard  copy  printouts  remains  to 
be  investigated. 
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CHAPTER  VIII 


INSTRUCTOR  CONSOLE  DESIGN  AND  LOCATION 


SUMMARY 

This  chapter  is  presented  in  two  parts.  One  deals  with  instructor 
console  design,  and  emphasis  is  placed  on  consoles  that  are  located 
remotely  from  the  student's  work  station.  References  are  made  to 
detailed  design  guidelines  presented  in  earlier  chapters  of  this  report. 
The  second  part  deals  with  the  issue  of  where  the  instructor's  station 
should  be  located  (at  or  near  the  student's  station  versus  at  a  remote 
location)  from  the  standpoint  of  how  the  student  likely  would  benefit 
from  each  alternative. 

CONSOLE  DESIGN 

Introduction 


It  is  quite  difficult  to  provide  generalized  guides  on  how  to  design 
ATD  instructor  consoles  without  reference  to  specific  ATD  training 
applications  and  without  a  reasonable  analysis  of  instructor  and 
operator  tasks  that  the  console  is  to  be  designed  to  support.  Preceding 
chapters  in  this  report  provide  design  guidelines  for  a  number  of 
individual  instructional  support  features.  The  reader  is  advised  also 
to  review  this  information  with  respect  to  console  design.  Large  design 
information  gaps  exist,  however,  on  how  to  effectively  combine  a  number 
of  individual  instructional  features,  displays  and  related  controls  into 
a  single  console  configuration.  This  is  particularly  the  case  when 
multiple  instructors  and  multiple  device  operators  must  work  effectively 
as  team  at  the  console.  Additional  research  is  needed  to  develop  guides 
for  the  design  of  large,  complicated,  multi-crew  consoles. 

Remaining  sections  of  this  chapter  deal  with  design  features  often 
overlooked  nrfien  designing  instructor  stations,  particularly  those  that 
are  remote  from  the  student's  workstation;  a  number  of  console  design 
issues  that  are  of  concern  because  of  recent  trends;  and  recommendations 
for  prototype  console  evaluation  before  conmitting  to  production. 

Overlooked  Features 


A  number  of  console  features  seem  to  be  overlooked  frequently  during 
design.  Some  are  at  the  nuisance  level,  but  they  are  presented  here 
with  the  intent  of  eliminating  such  nuisances  in  the  future. 

Instructor  consoles  frequently  do  not  have  adequate  work  space  for 
the  routine  paperwork  required  during  ATD  sessions.  There  seems  never 
to  be  adequate  surface  workspace  or  convenient  storage  space  for  maps, 
charts,  syllabi,  grading  forms  and  guides,  and  the  like.  Existing  human 
engineering  criteria  should  be  applied  in  this  respect. 
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The  communication  and  navigation  radio  frequencies  that  the  student 
has  tuned  to  in  the  ATD  cockpit  seldom  are  displayed  at  instructor 
consoles.  As  a  consequence,  the  instructor  frequently  must  contact  the 
student  on  the  ATD  intercom  to  seek  communication  and  navigation 
information.  A  positive  forward  step  would  be  to  display  selected 
conmunication  and  navigation  frequencies  at  the  instructor's  console. 

Instructor  information  needs  from  repeater  cockpit  instruments  may 
be  different  from  the  student's  needs.  Cases  were  observed  where 
electronic  repeater  displays  (typically  attitude  director  displays  and 
head  up  displays)  at  the  instructor's  console  were  slaved  to  modes 
selected  by  the  student  in  the  ATD.  Thus,  steering  command  information, 
lead  computing  site  information  and  similar  items  were  not  accessible  to 
the  instructor  unless  the  student  selected  them,  although  their  access 
could  have  assisted  the  instructor  in  evaluating  and  guiding  student 
learning  in  how  to  use  various  (e.g.  degraded)  display  modes.  In  other 
words,  what  the  instructor  needs  to  see  at  his  console  should  not  be 
limited  by  what  the  student  chooses  to  see  in  the  simulator. 

Virtually  every  ATD  remote  instructor's  console  surveyed  lacked 
plugs  for  extra  headphones,  so  that  other  instructors,  including 
instructor  trainees,  could  listen-in  conveniently  to  conversations 
between  the  ATD  instructor  and  the  student.  Extra  headphone  plugs  also 
prove  valuable  when  visitors  are  touring  ATD  facilities. 

No  ATD  surveyed  displayed  at  the  instructor's  console  information  on 
the  status  of  ATD  support  systems,  such  as  hydraulics,  electrical  and 
air  conditioning.  Whether  this  information  needs  to  be  displayed  must 
be  decided  on  a  case  by  case  basis  depending  on  the  tasks  that  console 
instructors  and  operators  are  to  perform  at  the  console.  At  issue  is 
the  tasks  that  are  assigned  to  ATD  instructors.  If  they  are  to  monitor 
ATD  systems,  then  their  displays  and  training  must  address  these  issues. 

Issues  of  Concern 

An  issue  of  continuing  concern  is  that  the  aircrew  instructor's  ATD 
job  seems  to  have  escaped  needed  detailed  analysis.  This  issue  is 
discussed  In  Chapter  IX,  Instructor/Operator  Training,  in  the  context  of 
an  Instructional  System  Development  (ISD)  approach  to  ATD  instructor 
training.  The  same  issue  is  of  concern  with  respect  to  console  design, 
because  consoles  are  pieces  of  equipment  meant  to  support  men  in  the 
doing  of  their  jobs.  It  is  only  reasonable,  therefore,  to  start  with  a 
thorough  knowledge  of  the  tasks  and  performance  the  instructor's  job 
requires  so  that  consoles  can  be  designed  to  support  these  requirements. 
It  probably  Is  the  case  that  at  least  some  ATD  manufacturers  take  the 
Instructor's  tasks,  sequences,  workloading  and  coordination  requirements 
Into  account  when  they  design  ATD  instructor  consoles.  What  is  needed, 
however,  are  Improved  guidelines  for  relating  Instructor  task 
requirements  to  console  design  characteristics  so  that  what  results  will 
be  more  predictably  workable. 
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Related  to  the  above  issue  is  a  driving  technology  force  in  ATD 
console  design.  Electronic  displays,  such  as  cathode  ray  tube  displays 
(CRT),  are  showing  up  in  increasing  numbers  in  ATD  instructor  consoles. 
By  themselves,  they  are  not  bad;  in  fact,  electronic  displays  provide 
many  advantages,  as  discussed  previously  in  this  report.  However,  the 
age-old  goal  of  simplicity  of  design  and  operation  is  easily  endangered 
by  unrestrained  use  of  this  technology.  Given  a  highly  versatile  display 
medium  such  as  the  CRT,  literally  hundreds  of  pages  of  information  can 
be  created  for  display.  The  technology  to  do  the  programming  is 
available.  The  technology  to  guide  the  programming  cannot  be  drawn 
solely  from  site  visits  involving  older  technology  devices  or  the 
existing  ATD  research  literature.  Thus,  a  lingering  concern  among  many 
training  specialists  is  that  instructor  workload,  training  requirements 
and  device  acceptance  will  suffer  unless  research  is  directed  toward 
developing  meaningful  guides  for  capitalizing  on  new  console  technology 
without  being  captured  by  it.  Air  Force  research  now  is  beginning  to 
address  this  issue. 

Taking  a  somewhat  different  view  point,  concern  also  is  evident  in 
the  training  community  that  instructor  consoles  that  are  located 
remotely  from  the  student  workstation  may  not  provide  instructors  with 
needed  student  performance  information,  or  may  provide  it  in  forms  that 
are  difficult  to  use.  One  issue  of  concern  is  how  best  to  display  for 
the  instructor  the  many  hundreds  of  discrete  event  procedures  (e.g. 
switch  positions  and  sequences)  that  students  perform.  Individual 
annunciator  lights  showing  discrete  switch  positions  can  be  so  numerous 
as  to  be  overwhelming  to  the  instructor.  Also,  the  annunciators,  at 
least  in  many  present  applications,  often  do  not  display  events  in 
natural  sequences.  Rather,  the  instructor  is  required  to  learn  patterns 
of  annunciator  illuminations  in  order  to  be  able  to  critique  student 
procedural  performance  and  provide  guidance.  An  alternative  is  to 
display  discrete  procedural  event  sequences  using  electronic  display 
technology.  This  is  being  done  in  a  number  of  new  ATDs  (e.g.  B-52, 
C-130,  A-10  and  F-16).  However,  this  also  can  result  in  a  burdensome 
instructor  monitoring  and  evaluation  task.  An  additional  alternative  is 
to  rely  on  an  automated  performance  measurement  system  to  monitor 
procedures  performance,  and  alert  the  instructor  only  to  discrepancies 
in  desired  performance.  The  issue  of  how  best  to  display  discrete  event 
procedural  performance  information  remains  an  important  issue  to  be 
resolved. 

A  further  issue  of  concern  is  the  nature  of  ATD  visual  scene 
information  needed  at  a  remote  instructor's  console  to  enable  productive 
training.  The  fidelity  volume  (pages  54  and  55)  for  example,  discusses 
the  possibility  that  one  versus  one  air  combat  training  was  not  possible 
In  one  Air  Force  ATD  for  students  unfamiliar  with  the  tasks,  because 
their  Instructors  did  not  have  adequate  visual  Information  on  student 
performance  or  the  adversary's  performance.  This  factor  probably  made 
ATO  Instruction  Impossible,  which  could  explain  why  experience  In  the 
ATD  resulted  in  no  improvement  in  airborne  performance.  Research  is 
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required  on  this  issue,  because  the  costs  of  mass-produced  failures 
should  be  measured  in  more  than  just  dollars. 

Recommendations 

This  section  has  presented  several  specific  recommendations  for 
research  that  are  not  repeated  here.  In  addition  to  the  above 

recommendations,  it  also  is  recommended  that  research  be  undertaken  to 
develop  design  guides,  particularly  for  more  complicated  instructor 
consoles,  so  that  their  utility  and  acceptance  can  be  ensured.  In 
addition,  until  programatic  research  is  completed,  it  is  recommended 
that  instructor  console  prototypes  be  developed  and  evaluated  before 
designs  are  frozen.  Related  to  this,  it  is  reconmended  that  prototype 
designs  and  evaluations  be  well  documented  so  that  lessons  learned  from 
them  are  available  for  future  use. 

CONSOLE  LOCATION 

Introduction 

The  issue  of  where  the  ATD  instructor's  console  (and  hence  the 
instructor)  should  be  located  physically  with  respect  to  the  student  has 
been  dominated  by  perceived  needs  for  exact  physical  correspondence 
between  the  ATD  cockpit  (student  workstation)  and  the  aircraft  cockpit. 
Typically,  if  the  instructor  can  be  present  in  the  aircraft  cockpit,  he 
can  be  present  in  the  ATD  cockpit.  If  not,  then  ATD  designs  have 
relegated  instructors  to  remote  console  locations.  Thus,  the  issue  has 
been  and  largely  remains  a  physical  system  fidelity  issue  rather  than 
a  training  issue.  Site  visit  survey  results  showed  that  instructors 
strongly  preferred  whatever  arrangement  it  was  that  they  already  had. 
Careful  analysis  of  this  trend,  in  combination  with  several  recent  ATD 
design  trends,  strongly  suggests  an  underlying  instructor  desire  to  be 
able  to  see  what  the  student  is  doing  so  that  he  can  better  provide 
training  feedback  and  guidance. 

Considerations  for  Onboard  Console  Location 

A  number  of  multi-crew  ATDs  were  observed  where  instructor  stations 
were  at  the  student  workstation  because  this  is  the  case  in  the 
aircraft.  In  these  cases,  Instructors  strongly  favored  onboard 
Instructor  station  locations.  A  number  indicated  that  they  did  not 
known  how  Instruction  could  be  accomplished  In  any  other  way.  When 
these  devices  Incorporated  remote  operator  consoles,  the  functions  of 
the  remote  consoles  typically  were  to  support  communication,  navigation 
or  electronic  warfare  training  activities.  Control  of  the  training 
simulation,  however,  could  be  accomplished  from  onboard  Instructor 
stations. 

Instructor  preference  for  being  onboard  centered  on  Improved 
Information  gathering.  For  example,  advantages  cited  Included  being 
able  to  observe  student  Instrument  scan  patterns,  visually  detect 


hesitations  in  physical  movements,  better  observe  crew  coordination,  and 
observe  how  the  students  used  checklists,  charts  and  other  technical 
information.  Instructors  also  indicated  improved  opportunities  to  guide 
students  in  how  to  perform  their  tasks,  and  provide  "fine  tuning" 
feedback  on  student  performance.  All  of  these  are  valid  instructional 
points. 

ATO  costs  usually  are  less  for  onboard  instructor  stations,  because 
the  instructor  can  observe  actual  cockpit  instruments,  controls,  and 
depending  on  its  design,  simulated  out  of  cockpit  visual  scenes.  They 
do  not  need  to  be  reproduced  elsewhere. 

Observations  suggest  that  the  perceived  need  for  a  remote 
instructor's  station  does  not  become  strong  until  training  in  flight 
control  and  mission  oriented  tasks  is  begun.  Then,  if  the  aircraft 
excludes  the  Instructor,  strong  feelings  develop  that  the  ATD  also 
should  exclude  the  Instructor.  However,  several  Interesting  exceptions 
exist. 

Two  ATDs  surveyed  included  closed  circuit  television  systems  so  that 
Instructors  at  remote  consoles  might  see  what  students  were  doing. 
Neither  was  used  because  of  technical  reasons.  However,  the  perceived 
need  for  their  initial  existence  suggests  that  cockpit  physical  fidelity 
was  being  adhered  to,  but  cheated  at  the  same  time  in  order  to  make 
direct  observations  of  student  behavior  possible. 

Procedures  trainers,  even  for  single  seat  aircraft  configurations, 
typically  allow  easy  instructor  visual  and  verbal  access  to  students. 
This  type  of  ATD  design  Is  well  accepted.  The  same  situation  was 
observed  for  a  weapon  system  trainer  (FB-111).  In  this  case, 
instructors  spoke  highly  of  being  able  to  directly  observe  and  work  with 
the  student  throughout  training.  During  performance  demonstration  runs, 
of  course,  the  Instructor  did  not  work  directly  with  the  student,  but 
acted  as  a  passive  observer.  It  Is  easy  to  argue  that  single  seat  ATDs 
for  pilot  training  should  have  Instructor  observation  stations  In  the 
cockpit  for  the  same  reason. 

By  way  of  suwnary,  there  are  many  Instructional  advantages  to  having 
the  Instructor  colocated  with  the  student,  even  if  this  means  a 
departure  from  strict  adherence  to  the  physical  fidelity  of  the  cockpit. 
However,  a  strong  emotion  exists  to  allow  Instructors  to  be  colocated 
only  if  that  Is  the  case  In  the  aircraft,  and  particularly  when  flight 
control  and  mission  training  are  undertaken. 

Considerations  for  Remote  Console  Location 

Remotely  located  Instructor  consoles  were  used  commonly  when  the  ATD 
simulated  an  aircraft  that  could  not  accommodate  an  Instructor  In  an 
observer  type  position  during  flight.  Thus,  most  single  seat  and  two 
seat  simulations  had  remote  Instructor  consoles. 
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Instructors  using  these  devices  argued  strongly  that  they  should  not 
be  colocated  with  the  student  because  this  was  not  possible  In  the  air. 
A  frequently  encountered  argument  was  that  Instructors  did  not  want 
students  to  learn  to  count  on  an  Instructor's  presence  when  this  would 
not  be  possible  during  flight.  The  argument  that  Instructors,  on  some 
occasions,  must  not  be  colocated  with  students  hinges  on  the  physical 
fidelity  Issue  and,  of  course.  Impacts  on  the  design  and  cost  of  remote 
Instructor  consoles.  The  argument  also  side-steps  the  fact  that  ATDs 
are  training  tools,  rather  than  weapon  systems.  All  training  tools 
should  be  designed  to  promote  effective,  efficient  training. 

Research  Evidence 

Only  very  scant  research  evidence  exists  specifically  on  the  Issue 
of  Instructor  station  location.  Existing  evidence  favors  an  onboard 
location. 

The  evidence  comes  from  comparing  two  simulator  studies  involving 
the  training  of  pilots  unfamiliar  with  air  combat  maneuvering  (ACM)  on 
one  versus  one  ACM.  One  study  (Payne,  et  al.,  1976)  showed  that 
simulator  training  transferred  positively  to  inflight  performance.  The 
simulator  they  used  had  a  hemispherical  dome  projection  system,  so  an 
Instructor  could  be  (and  was)  seated  beside  the  student,  even  though  the 
aircraft  simulated  was  an  F-4.  Pohlman  and  Reed  (1978),  however,  found 
no  Inflight  benefit  from  comparable  simulator  training.  The  device  they 
used  was  the  Air  Force  Simulator  for  Air  to  Air  Combat  (SAAC),  which 
also  Is  an  F-4  simulation.  This  device  used  a  direct  view  visual  system 
with  infinity  optics.  Although  the  imagery  was  more  realistic  (i.e. 
higher  fidelity),  instructors  were  unable  to  view  what  the  student  was 
seeing  or  doing.  Thus,  they  could  not  directly  provide  guidance  or 
feedback.  No  training  took  place  In  the  simulator,  and  as  expected  no 
performance  improvement  was  observed  in  the  air.  It  is  a  reasonable 
possibility  that  instructors  may  have  to  be  colocated  with  students  for 
ATD  training  on  at  least  some  types  of  tasks,  particularly  during  early 
stages  of  learning.  If  they  are  not  colocated,  they  may  have  to  be 
provided  with  very  extensive  and  costly  information  at  remote  consoles 
in  order  to  provide  meaningful  training. 

The  U.S.  Army  recently  procured  a  training  simulator  for  a  two-place 
attack  helicopter.  This  simulator  is  unique  in  that  the  pilot's  station 
is  separate  from  the  gunner's  station.  Each  separate  station,  however, 
was  designed  as  a  two-seat  configuration  so  that  Instructors  could  be 
present  in  the  simulation  during  training.  This  device  was  surveyed, 
but  before  experience  with  It  in  training  had  been  gained.  A  follow-up 
certainly  is  warranted. 

An  Alternative 

The  Army  rotary  wing  training  device  described  above  provides  one 
option.  Another  alternative  is  to  design  ATDs  so  that  Instructors  can 
be  present  when  needed,  and  not  present  when  this  is  better.  For 
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example,  an  instructor's  station  In  a  single  seat  ATO  could  be  located 
behind  and  to  one  side  of  the  student.  The  instructor,  his  controls  and 
displays  would  be  quite  unobtrusive.  Yet,  he  could  observe  student 
performance  directly,  and  provide  guidance  and  feedback  as  required. 
Performance  monitoring  displays  also  could  be  incorporated  in  a  remote 
console  so  that  the  instructor  could  monitor  student  performance  there 
during  evaluation  runs.  During  these  runs,  a  canopy  could  be  closed 
around  the  student's  portion  of  the  ATD  to  promote  realism.  Also, 
nothing  would  be  sacrificed  in  important  fidelity  areas  such  as  the 
physical  location  of  cockpit  controls  and  displays. 
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CHAPTER  IX 


INSTRUCTOR/OPERATOR  TRAINING 


SUMMARY 

ATO  instructors  and  operators  are  central  to  the  effective  and 
efficient  use  of  these  devices  for  aircrew  training.  Part  of  the 
instructor  and  operator's  job  is  knowing  how  to  operate  ATDs.  A 
significant  part  of  the  instructor's  job  is  knowing  how  to  use  ATDs  and 
their  capabilities  to  teach  effectively.  The  program  survey  showed  that 
a  vast  majority  of  instructor  training  deals  with  how  to  do  the  tasks  he 
is  to  train,  safety,  and  training- related  administrative  matters.  On 
the  average,  only  about  three  hours  of  formal  instruction  dealt  with  how 
to  be  a  teacher.  In  virtually  all  cases  reviewed,  instructor  and 
operator  training  on  how  to  operate  ATDs  was  left  to  informal  on  the  job 
training.  Instructor  training  on  how  meaningfully  to  use  ATDs  and  their 
instructional  features  was,  again,  left  almost  exclusively  to  informal 
on  job  training  with  no  form  of  quality  control.  The  STRES  report 
titled  Utilization  of  Aircrew  Training  Devices  addresses  the  general 
issue  of  ATD  instructor  training  in  Chapter  VI.  Numerous  other  chapters 
in  the  Utilization  report  deal  with  principles  of  effective  instruction 
in  ATDs.  The  balance  of  this  chapter  deals  with  instructor  training  on 
effective  use  of  ATD  instructional  support  features,  and  is  meant  to 
complement  the  content  of  the  utilization  report. 

BACKGROUND 

The  best  of  training  equipment,  by  itself,  will  not  produce 
operationally  ready  aircrews.  The  equipment  must  be  used  effectively  to 
achieve  this  goal.  Obviously,  instructor  and  operator  training  is 
central  to  the  effective  and  efficient  use  of  ATDs  for  aircrew  training. 
The  fact  remains,  however,  that  virtually  every  Air  Force  aircrew 
instructor  training  program  focuses  on  how  to  perform  the  tasks  to  be 
trained,  safety  of  flight  considerations,  and  administrative  facets  of 
student  review  and  management.  These  are  important  issues.  However, 
information  gathered  during  this  program  suggests  that  each  Air  Force 
instructor  pilot  receives  only  about  three  hours  of  formal  instruction 
on  how  to  be  a  teacher.  This  seems  very  little  when  one  considers  the 
educational  background  and  training  the  nation  demands  of  public  school 
teachers,  for  example. 

RECOMMENDATIONS 

Better  Instructor  training  almost  certainly  is  an  area  where 
Improved  training  cost  and  student  performance  benefits  would  result. 
To  date,  the  aircrew  Instructor's  job  has  been  examined  only  In  general 
terms.  The  result  has  been  descriptions  of  the  instructional  and 
administrative  functions  performed  by  ATD  insf 


for  ATO  design  and  use  based  on  job  function  descriptions.  To  date, 
however,  no  one  has  taken  a  detailed,  task-level  look  at  a  spectrum  of 
aircrew  instructor  jobs  to  identify  specifically  what  it  is  they  do, 
what  could  be  done  even  better,  and  what  task  facets  are  common  versus 
unique  to  different  training  applications.  In  short,  the  first  detailed 
Instructional  System  Development  (ISD)  step  has  not  been  taken  for  the 
most  basic  facet  of  an  instructor's  job  —  i.e.  teaching.  It  is 
recommended,  therefore,  that  a  systematic  analysis  of  ATD  instructor 
tasks  be  undertaken  so  that  improved  instructor  training  can  be 
objectively  related  to  required  job  performance. 

Analysis,  of  course,  is  a  means  rather  than  an  end;  and  analyses 
require  a  certain  time  investment.  A  reasonable  question,  then,  is  what 
can  be  done  until  the  needed  analyses  are  performed?  Some  specific 
guidance  is  available  for  Instructor  training  on  ATD  Instructional 
support  feature  use.  Many  sections  of  chapters  of  this  volume  have 
addressed  this  issue.  For  example,  guidance  is  presented  on  common  and 
potential  instructional  feature  uses.  Also,  information  on  observed 
feature  applications  has  been  presented  to  further  stimulate  creative 
thought  on  how  to  better  use  available  features  in  existing  ATDs. 
Finally,  utility  information  sections  often  present  further  suggestions, 
together  with  the  experiences  of  others  with  various  instructional 
support  features.  This  information,  together  with  the  content  of 
companion  STRES  reports,  can  provide  a  nucleus  for  instructor  seminars 
or  other  types  of  training  on  how  to  use  many  ATD  capabilities  and 
features.  It  is  recommended,  therefore,  that  steps  be  taken  to  transfer 
to  Instructors  the  information  now  available  to  them  in  this  and 
companion  STRES  program  reports.  It  is  further  recommended  that  Air 
Force  aircrew  instructor  training  be  systematically  reviewed  and  revised 
to  better  match  Instructor  training  with  Instructor  job  requirements. 
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APPENDIX  A 


INSTRUCTIONAL  SUPPORT  FEATURES 
INTERVIEW  GUIDE 


The  interview  guide  presented  in  this  appendix  was  used  to  guide 
interviews  conducted  during  program  site  visits  with  respect  to  ATD 
instructional  support  features.  An  interview  guide  is  a  checklist  of 
topics,  to  be  addressed.  Its  purpose  is  to  guide  and  structure  indepth, 
penetrating  interviews.  Unlike  a  questionnaire,  which  is  highly 
structured,  an  interview  guide  does  not  exclude  meaningful  avenues  of 
investigation  that  suggest  themselves  during  the  course  of  data 
gathering.  A  primary  use  of  the  guide  in  this  appendix  was  a 
checklist,  which  was  referred  to  at  the  close  of  each  day  of 
interviewing  to  assure  that  all  relevant  topics  were  covered  during  each 
site  visit. 
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INSTRUCTIONAL  SUPPORT  FEATURE  INVENTORY 


1.  Event  Definition  and  Control 

a.  Pre-session  audio-visual  demonstrations 

b.  Simulator  initialization  procedures 

c.  Automated  demonstration s) 

d.  Pre-programmed  (canned)  mission  events 

e.  On-line  progranmabl e  mission  events 

f.  On-line  capability  for  establishing  event  initial  conditions 

g.  Auto  mal function/ emergency  insertion-removal;  and  capability  to 
add  additional  malfunctions/emergencies 

h.  Manual  malfunction/emergency  i nsertion- removal ;  and  capability 
to  add  additional  malfunctions/emergencies 

i.  Dedicated  controls  (functional  groupings) 

j.  Keyboards 

k.  Joy  sticks  and  related  controls 

l.  CRT  data  entry/ select ion  formats  (touch  panel,  other) 

m.  Threat  control 

n.  Environment  control  (rough  air,  etc.) 

o.  Task  difficulty  control 

p.  Freeze 

q.  Reset 

r.  Record  and  replay 

2.  Event  Monitoring 

a.  Repeater  Instruments  (functional  groupings) 

b.  Electronic  pictorial /analog  display(s) 

c.  Other  performance  and  event  monitoring  displays 

d.  Mechanical  plotter 
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e.  Closed  circuit  TV,  and  record/playback  capabilities 

f.  Other  video  monitoring  capabilities,  and  record/playback 
capabilities 

g.  Hard  copy  output 

3.  Instructor/Operator  -  Student  Communication 

a.  Usual  methods  of  communicating  with  student  work  station 

b.  Cockpit-mounted  alphanumeric  displays 

c.  Cockpit-mounted  responder  controls 

d.  Computer  generated  voice 

e.  Automated  speech  understanding 

f.  Simulator-mounted  audio-visual  instructional  devices 

4.  Automated  Support 

a.  Automated  performance  measurement 

b.  On-line  performance  monitoring  and  alerting 

c.  Automated  scoring 

d.  Automated  learning  problem  diagnosis 

e.  Automated  coaching 

f.  Automated  cuing 

g.  Automated  performance  norm  data  files 

h.  Computer  managed  instruction 
INSTRUCTIONAL  SUPPORT  FEATURE  SURVEY  ELEMENTS 
1.  Description 

a.  Manufacturer  and  date 

b.  Functional  description 

c.  Modes  of  operation 

d*  Man-machine  Interface 
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e.  Representative  operating  procedures 

f.  Difficulty  of  use 

g.  Factors  affecting  operability  (ease/ complexity  and  time) 

h.  Planned  design  changes 

i.  Desirable  design  changes 

j.  Local  modifications  made  and  date  (hardware  and  software) 

k.  Contracted  modifications  and  date 

l.  Reasons  for  modifications 

m.  Perceived  benefits  from  modifications 

n.  Training  evaluations  of  modifications 

o.  Surveyor's  assessments 
Training  Applications 

a.  Present  training  applications  (tasks) 

b.  Training  content  involved  (tasks) 

c.  Specific  student  learning  and/or  performance  problems 

d.  Instructional  uses  originally  designed  to  support 

e.  Reasons  for  additions/deletions  to  uses 

f.  Training  evaluations  performed 

g.  Planned  changes  to  training  that  could  affect  applications 

h.  Suggestions  for  other  applications 
1.  Surveyor's  assessment 
Utilization 

a.  Job  performance  aids  and  user  assists  available 

b.  Perceived  values  of  the  feature 

c.  Perceived  drawbacks  of  the  feature 
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d.  Frequency  of  use 

e.  Reliability 

f.  Procedures  and  forms  to  report  maintenance  problems 

g.  Factors  influencing  feature  utilization 

(1)  Training  requirements 

(2)  Perceived  benefits 

(3)  Perceived  drawbacks 

(4)  Design/implementation 

(5)  Schedule  considerations 

(6)  Instructor  training 

(7)  Attitudes  toward  simulation 

(8 )  Other 

h.  Characterize  impacts  of  feature  on  ATD  utilization 

i.  Essentiality 

Cite  representative  uses  of  the  feature  that  fall  into  the 
f ol 1 owi ng  categor i es . 

(1)  Highly  essential.  The  feature  is  required  to  accomplish 
simulator  training. 

(2)  Moderately  essential.  The  feature  aids  in  accomplishing 
simulator  training,  but  it  would  be  possible  to  get  along 
without  it. 

(3)  Marginally  essential.  The  feature  is  convenient  for 
simulator  training,  but  it  would  be  easy  to  get  along 
without  it. 

(4)  Totally  non-essential. 

J.  Surveyor's  assessments 


1 


158 


SURVEY  ELEMENTS  UNIQUE  TO  CERTAIN  FEATURES 

1.  Record  and  Replay 


a.  Where  is  replay  accomplished? 

(1)  in  cockpit 

(2)  remote  (location  and  content) 

(3)  availability  of  augmented  feedback  during  replay  (e.g. , 
scoring  or  diagnostics) 

b.  How  much  of  the  mission  can  be  replayed? 

2.  Freeze 

a.  Can  freeze  be  operated  during  replay? 

3.  Video  Display 

a.  Description  (field  of  view,  size,  resolution,  color,  number) 

b.  Record/replay  capabilities 

4.  Automated  Demonstrations 

a.  Capabilities  and  procedures  for  constructing/modifying  canned 
scenarios 

(1)  On-line  (who  and  how) 

(2)  Off-line  (who  and  how) 

b.  Difficulties  In  developing  and  using  canned  scenarios 

(1)  Development 

(2)  Call-up  and  use 

5.  Automated  Performance  Measurement 
a.  Descriptive  data 

(1)  Manufacturer  and  delivery  date 

(2)  Add-on  to  simulator? 

(3)  Production  or  RAD  feature/capability? 

(4)  Describe  any  In-house  modlficlations 


(5)  Describe  any  on-going  A PM  R4D 

(6)  Obtain  system  and  RAD  documentation 

(7)  Identify  RAD  point  of  contact  (name,  organization, 
telephone) 

b.  Characteristics 

(1)  Parameters  that  can  be  measured  (total  set) 

(2)  How  are  the  parameters  designated  (procedures)? 
on-line 

off-line 

(3)  How  are  performance  tolerances  set? 
on-line 

off-line 

(4)  How  is  measurement  start  and  stop  designated? 
on-line 

off-line 

(5)  Are  any  mathematical  transforms  performed? 

(6)  How  and  when  are  the  measures  displayed?- 

(7)  Utility  of  measures  for  quantifying  proficiency  (criteria 
used)  (use  of  measurement  outputs  when  completing  student 
grade  forms?) 

(8)  Diagnostic  value  of  the  measures 

(9)  Utility  of  the  measurements  for  training  quality  control 

(10)  Desirable  improvements 

(11)  Other  applications 


Automated  Scoring 

a.  Describe/define  the  combining  of  measures  into  scores 

b.  Describe/define  scoring  criteria  used 


c.  Describe  Intended  uses  of  the  scores 

d.  Utility  of  performance  scores  for  training  quality  control 

e.  Use  of  scores  when  completing  student  grade  forms 
Automated  Adaptive  Training  (AAT) 

a.  Descriptive  Data 

(1)  Manufacturer  and  delivery  date 

(2)  Add-on? 

(3)  Production  or  RAD  feature/capability 

(4)  Describe  any  in-house  modifications 

(5)  Describe  any  related  RAD  activities 

(6)  Obtain  system  and  RAD  documentation  , 

(7)  Identify  RAD  point  of  contact  (name,  organization, 
telephone) 

b.  Characteristics 

(1)  Describe  adaptive  logic  structure(s)  used 

(2)  Describe  automated  measurement/ scoring  used 

(3)  Describe  on-site  modifications  and  reasons 

(4)  Training  Impacts  of  modifications 

(5)  Effects  of  AAT  system  on: 

(a)  Student  progress  through  automated  portions  of 
syllabus 

(b)  Student  proficiency 

(c)  Training  quality  control  (portions  of  syllabus 
Involved) 

(6)  Effect  of  AAT  on  Instructor  Involvement 

(7)  Effect  of  AAT  on  operator  Involvement 
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(8)  Student  acceptance  of  AAT 

(9)  Instructor  acceptance 
8.  Computer  Managed  Instruction 

a.  Descriptive  Data 

(1)  Name  and  acronym 

(2)  Developer  and  installation  date 

(3)  Cognizant  organization  and  symbol 

(4)  Obtain  documented  description  of  the  system 

(5)  Training  supported  (academics,  simulator,  inflight) 

b.  Characteristics 

(1)  Describe  how  it  is  used  (general) 

(2)  Input  data  sources 

(3)  Simulator  training  objectives  tracked 

(4)  Inflight  training  objectives  tracked 

(5)  Utility  in  determining  simulator  session  training  content 

(6)  Utility  in  determining  flight  training  content 

(7)  Utility  in  scheduling  simulator  sessions 

(8)  Utility  in  scheduling  training  flights 

(9)  Impacts  on  training  quality  control 
INSTRUCTOR/OPERATOR  STATION  DESIGN 

1.  Present  configuration 

a.  Device  designation  and  type 

b.  Linked  mode  operation? 

c.  Locatlon(s)  relative  to  student  workstation  v 

d.  Present  opportunities  for  dynamic  observation 

e.  Functions  performed  (at  each  location) 


f.  Instructors/operators  assigned  to  each  location 

g.  Instructor/operator  interaction 

h.  Ease/difficulty  of  coordination 

i.  Perceived  benefits  of  present  configuration 

j.  Perceived  drawbacks  of  present  configuration 

k.  Suggested  design  modifications  (and  reasons) 

2.  General  Perceptions  and  Suggestions 

a.  Types  of  training  where  direct  observation  is  perceived  as 
necessary 

b.  Types  of  training  where  direct  observation  is  perceived  as 
desirable 

c.  Student  skill  level  impacts  on  instructor/ operator  station 
locations 

d.  Device  type  (CPT,  OFT,  WST,  etc.)  impacts  on 
instructor/operator  station  locations 

e.  Perceived  general  benefits  and  drawbacks  of  in-cockpit  location 

f.  Perceived  general  benefits  and  drawbacks  of  remote  location 

g.  Advantages/disadvantages  of  designing  student  workstations  to 
Incorporate  an  instructor/operator  station  if  it  meant  that 
the  student's  workstation  environment  might  not  appear  exactly 
as  in  the  aircraft 

3.  Background 

a.  Interviewee's  prior  experience  with  remote  and/or  in-cockpit 
instructor  stations. 

(1)  As  a  student  (weapon  system,  device,  dates) 

(2)  As  an  Instructor  (weapon  system,  device,  dates) 

DEVICE  OPERATIONS 

1.  Surveyor's  Assessments 

a.  Instructor  status/morale 


163 


b.  Adequacy  of  instructor  training 

c.  Problems  noted 

d.  Desirable  points  noted 

e.  Implications  for  other  programs 

2.  Typical  Device  Operator  Characteristics 

a.  Rank 

b.  Years  in  service 

c.  Number  of  months  at  this  assignment 

d.  Duration  of  this  assignment  in  months 

e.  Prior  device  operator  experience 

f.  Operator  selection  criteria/procedures 

g.  Continuity  In  operator  staff 

3.  Oevice  Operator  Job  Characteristics 

a.  Hours  worked  per  week,  training  support  activities 

b.  Hours  worked  per  week,  other  activities 

c.  Device  utilization  activities  description 

d.  Other  activities  description 

4.  Device  Operator  Training 

a.  Schools  attended 

b.  Factory  training  for  this  job 

c.  Local  program  for  operator  training 

(1)  Topics  covered 

(2)  Program  description 

(3)  Program  duration 

(4)  Role  of  formal  Instruction 

(5)  Role  of  on-the-job  training 
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(6)  OJT  supervision 

(7)  Criteria  for  completing  training 

d.  Perceived  value  of  school  training  for  this  job 

e.  Perceived  value  of  factory  training  for  this  job 

f.  Perceived  value  of  local  training  for  this  job 

g.  Planned  changes  in  operator  training  program 

h.  Suggested  changes  to  operator  training  program 
Surveyor's  Assessments 

a.  Operator  status/morale 

b.  Adequacy  of  operator  training 

c.  Problems  noted 

d.  Desirable  points  noted 

e.  Implication  for  other  programs 
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APPENDIX  B 

GUIDANCE  FOR  THE  DEVELOPMENT  OF  AUTOMATED 
PERFORMANCE  MEASUREMENT  SYSTEMS 


INTRODUCTION 


Automated  performance  measurement  (APM)  systems  require  a  deliberate 
development  procedure  if  they  are  to  be  useful  and  accepted.  This 
appendix  presents  general  specifications  for  APM  systems  for  ATDs  and 
then  presents  a  workable  APM  development  procedure. 

Functional  Specifications  for  AMP  Systems 

The  following  functional  specifications  have  been  developed  for 
users  who  have  need  to  specify  an  APM  system  as  part  of  a  new  ATD 
procurement  or  the  retrofitting  of  existing  ATD.  They  have  been 
developed  from  recent  APM  research  and  development  work  (See  the  Related 
Research  section  in  the  APM  portion  of  Chapter  III  of  this  report). 

Level  1  APM  System.  (Continuation  Training  Profiency  Assessment) 

Purpose.  To  provide  a  system  that  will  measure  and  score  the 
exceeding  of  criteria  prescribed  for  specific  operational  flight 
segnents  or  procedures. 

Application.  Continuation  training  proficiency  assessment  of  a 
pilot  who  has  been  qualified  previously  on  the  aircraft  type  or  in 
the  ATD  in  which  the  assessment  is  to  be  performed. 

Scope  of  Measurement.  To  automatically  measure  performance  on 
ground  referenced,  air  referenced,  and  proponent/opponent  referenced 
flight  profiles  and  other  specialized  missions  required  by  the 
continuation  training  curriculum. 

Method  of  Measurement.  Monitoring  of  selected  flight  and  control 
parameters  on  a  sampled  basis,  including  the  monitoring  of  time 
related  procedural  or  discrete  events  (e.g.,  landing  gear  lever 
down,  or  #1  hydraulic  pump  not  switched  off). 

Segmentation  of  Flight  Profiles.  Each  flight  profile  to  be 
automatically  tracked  using  stop/start  logic  based  on  discrete 
events  or  change  of  state  of  one  or  more  measured  parameters  to 
determine  the  beginning  and  end  of  each  segment  or  sub-segment  of 
the  profile. 

Method  of  Scoring.  Scoring  algorithms  to  be  based  on  exceeding  of 
limits  establish^  for  the  parameters  used  for  measurement  in  each 
specific  segment  of  a  flight  profile.  Separate  scoring  methods  are 
required  for  flight  and  control  measurement,  and  time  related 
discrete  events  chains.  The  partial  scores  so  obtained  should  then 
be  combined  using  empirically  derived  weighting  coefficients  into 
performance  scores  for  the  flight  segment,  overall  flight  profile, 
or  overall  recurrent  proficiency  assessment  as  required. 


Development  of  Score  Weightings.  A  primary  objective  is  to  arrive 
at  a  minimum  set  o^  measures  which  captures  the  performance  variance 
of  at  least  67  percent  of  the  pilot  population  used  in  developing 
the  performance  norms.  Initially,  the  scoring  algorithms  may  be 
based  on  information  obtained  from  highly  qualified  check  pilots  who 
normally  score  each  specific  performance  objective.  Subsequently, 
the  scoring  algorithms  must  be  refined  commensurate  with  a  “test" 
group  of  skilled  pilots  already  endorsed  on  the  specific  aircraft 
type  and  maneuvers  for  which  the  AID  is  used.  For  scoring  a  time 
related  discrete  event  chain  (for  example,  a  hydraulic  failure 
requiring  the  isolation  of  some  hydraulic  elements),  scoring  must 
take  into  account  whether  correct  outcome  was  achieved,  the  time 
taken  in  arriving  at  the  outcome,  and  that  an  optimum  discrete  event 
path  through  alternative  event  sequences  was  taken. 

Method  of  Presenting  Scores.  By  CRT  or  line  printer.  The 

achieved  scores  should  be  broken  down  to  show  each  partial  score  and 

how  it  compares  to  the  norm  established  for  the  task  being  assessed. 

Compatibility  of  ATP  APM  System  to  the  Aircraft.  Even  though  many 
ATDs  have  different  flight  control  characteristics  than  the 
aircraft  they  represent,  a  major  design  objective  for  the  ATD  APM 
system  is  to  approach  equality  with  the  scoring  appropriate  for 
pilot  performance  In  the  aircraft. 

Development  and  Validation.  This  should  be  an  orderly  process  in 

accordance  With  the  steps  shown  in  Figure  B-l.  The  process  Is 

further  described  later  in  this  appendix. 

Level  2  APM  System.  (Undergraduate  and  Transition  Training  Proflcency 

Assessment) 

Purpose.  To  provide  a  system  that  will  measure  and  score 
performance  when  measured  against  criteria  for  a  specific 
undergraduate  or  transition  training  program. 

Application.  Incremental  proficiency  assessment  of  a  pilot  who  is 
undergoing  training  (undergraduate  or  transition)  in  a  specific 
simulator  (or  aircraft)  In  which  the  assessment  is  taking  place. 
The  Level  2  system  appllction  may  Include  the  use  of  performance 
measurement  as  the  primary  control  element  In  an  automated  adaptive 
training  system  ttfilch  will  advance  the  pilot  through  a  flight 
training  curriculum  In  the  most  efficient  manner.  (See  Chapter  V  of 
this  report) 

Scope  of  Measurement.  To  automatically  measure  performance  on 
ground  referenced,  air  referenced,  and  proponent/opponent  referenced 
flight  profiles  and  other  specialized  missions  required  by  the 
training  curriculum. 
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Figure  B-l.  Automated  Performance  Measurement 
Design  and  Development  Process 
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Method  of  Measurement.  Monitoring  of  selected  flight  and  control 
parameters  on  a  sampled  basis  Including  the  monitoring  of  time 
related  procedural  or  discrete  events  (e.g.t  gear  lever  down,  or  #1 
hydraulic  pump  not  switched  off).  Depending  on  the  automated 
adaptive  training  requirements  for  the  measurement  system, 
interactive  aerodynamic  measurement  may  be  required  to  describe 
performance  In,  say,  a  climbing  turn.  Typical  aerodynamic  parameters 
measured  in  climbing  turn  would  be:  airspeed,  heading,  altitude 
change,  angle  of  attack,  side  slip,  power  setting,  stick  forces, 
stick  displacements,  pitch  and  roll  angles,  and  their  rates. 

Segmentation  of  Flight  Profiles.  Each  flight  profile  to  be 
automatically  tracked  using  stop/start  logic  based  on  discrete  event  or 
change  of  state  of  one  or  more  measured  parameters  to  determine  the 
beginning  and  end  of  each  segnent  and  sub-segnent  of  the  profile. 

Method  of  Scoring.  Scoring  algorithms  to  be  based  on  criterion 
values  for  critical  parameters  used  in  specific  segments  of  each 
flight  profile.  Separate  scoring  methods  are  required  for  flight 
and  control  measurements,  and  time  related  discrete  event  chains. 

The  "partial"  scores  so  obtained  should  then  be  combined  using 
empirically  derived  weighting  coefficients  into  separate  task  scores 
as  well  as  single  session  scores  for  diagnostic  and  evaluation 
purposes. 

Development  of  Score  Weightings.  A  minimum  set  of  measures  which 
captures  the  performance  variance  of  at  least  67%  of  the  pilot 
population  Is  a  primary  objective.  Initially  the  scoring  algorithms 
may  be  b?sed  on  Information  obtained  from  highly  qualified 
instructor  pilots  who  normally  score  each  specific  Incremental 
training  objective.  Subsequently,  the  scoring  algorithms  must  be 
refined  commensurate  with  a  representative  group  of  pilots  being 
trained  with  varying  degrees  of  experience  on  the  training  tasks  for 
which  the  ATD  Is  used.  For  scoring  a  time  related  discrete  event 
chain  (for  example,  a  hydraulic  failure  requiring  the  isolation  of 
some  hydraulic  elements)  scoring  must  take  Into  account  that  the 
actual  correct  outcome  was  achieved,  the  time  taken  In  arriving  at 
the  outcome,  and  that  an  optimum  discrete  event  path  through 
alternative  event  sequence  was  taken. 

Method  of  Presenting  Scores.  By  CRT  or  line  printer.  The  achieved 
score  should  be  broken  down  to  show  each  partial  score  and  how  It 
compares  to  training  criteria  established  for  that  segment  of 
flight. 

Compatibility  of  ATD  ARM  System  to  the  Aircraft.  Even  though  many 
Atds  have  different  flight  control  characteristics  than  the  aircraft 
they  represent,  a  major  design  objective  for  the  ATD  APM  system  Is 
to  approach  the  scoring  appropriate  for  pilot  performance  In  the 
aircraft. 


Development  and  Validation*  This  should  be  an  orderly  process  in 
accordance  with  the  steps  shown  previously  in  Figure  B-l.  The 
process  is  further  described  in  the  next  section  of  this  appendix. 

APM  DEVELOPMENT  PROCEDURE 

Prior  research  has  shown  that  twelve  general  steps  still  are 
required  to  develop  and  validate  a  workable  APM  system.  The  steps  are 
presented  below.  (See  the  Related  Research  section  of  the  APM  portion 
of  Chapter  III  of  this  report.) 

1.  Detailed  front  end  analysis  of  the  tasks  to  be  measured, 
including  the  selection  of  measures  considered  by  knowledgeable 
analysts  to  describe  profiency  In  those  tasks. 

2.  Design  and  development  of  a  data  collection  system  to  be  used 
In  the  operational  environment  to  establish  how  the  selected 
measures  perform  In  the  real  world. 

3.  Validation  of  the  data  collection  system  to  ensure  that  It 
produces  useable  data. 

4.  Collection  of  data  across  a  spectrum  of  aircrew  members  of 
varying  skill  levels  to  establish  a  "group"  baseline  of 
performance. 

5.  Initial  data  "screening"  analysis  (editing)  to  eliminate 
extraneous  data  which  could  distort  final  results. 

6.  Statistical  analysis  of  the  edited  data  using  multivariate 
statistical  techniques. 

7.  Selection  of  measures  which  describe  and  discriminate  observed 
performance  of  the  "group"  (usually  the  smallest  possible  set). 

8.  Empirical  development  of  scoring  algorithms  to  produce  single 
scores  for  the  tasks  measured.  This  requires  the  combining  and 
weighting  of  measures  from  sub- tasks. 

9.  Initial  validation  demonstration  using  Instructional  and 
training  crews  to  validate  the  objectively  generated  scores. 

10.  Collection  of  Instructor  pilot  ratings,  APM  scores  and  summary 
measurement  data  for  a  reasonable  sample  of  student  and 
experienced  pilots  to  provide  final  validation  of  the  system. 

11.  Release  of  the  system  for  training  use. 

12.  Continued  recording  of  measurement  data  for  APM  research  and 
system  refinement. 
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All  12  steps  presently  are  required  because  of  the  complexity  of 
measuring  aircrew  performance,  and  the  need  to  further  understand  these 
complexities  for  useful  measurement  system  development. 

MEASUREMENT  MODEL 

A  generalized  measurement  model  simply  is  a  systematic  way  of 
defining  automated  measures  that  uniquely  describe  measurements  for 
research  or  training  applications.  A  workable  model  that  defines  each 
measure  in  terms  of  the  following  five  determinants  has  been  shown  to  be 
practical  {Vreuls  and  Wooldridge,  1977).  Each  determinant  is  defined  In 
Chapter  IV  of  this  report.  Definitions  are  sunmarlzed  below. 

1.  A  measure  segment  (or  task) 

2.  A  state  variable 

3.  A  sampling  rate 

4.  A  desired  value  (If  required) 

5.  A  data  transformation 

A  segment  (or  task)  Is  any  portion  of  a  flight  for  which  desired 
behavior  or  system  performance  follows  a  lawful  relationship  from 
beginning  to  end,  and  for  which  the  beginning  and  end  can  be  defined 
unambiguously.  Measurement  start  and  stop  conditions  define  a  segment. 
Conditional  tests  of  ongoing  performance,  such  as  the  following,  are 
required  to  start  and  stop  measurement: 

If  (1)  Altitude  Is  Greater  Than  .Or.  Less  Than  1000  feet 
from  the  initial  value, 

•And . 

(2)  Heading  Is  Greater  Than  90  degrees  .And.  Less  Than 
270  degrees , 

Then,  start  measuring. 


A  variable  Is  any  quantitative  Index  of;  1)  vehicle  states  In  any 
reference  plane;  2)  personnel  physiological  states;  3)  control  device 
states;  or  4)  discrete  events.  A  sampling  rate  is  the  temporal 
frequency  at  which  the  variable  Is  examined.  Often,  variables  have  no 
utility  unless  compared  to  a  desired  value  or  a  tolerance  to  derive  an 
error  score.  Finally,  a  transformation  Is  any  mathematical  treatment  of 
the  parameter,  to  Include  measures  of  central  tendency,  variability, 
scalar  values,  Fourier  transforms,  pllot/system  transfer  functions,  etc. 
Common  measure  transforms  are  Identified  In  Table  B-l. 
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Table  B-l.  Comnon  Measure  Transforms 


Time  History  Measures 
Time  on  target 
Time  out  of  tolerance 
Maximum  value  out  of  tolerance 
Response  time,  rise  time,  overshoot 
Frequency  domain  approximations 

Count  of  tolerance  band  crossings 
Zero  or  average  value  crossings 
Derivative  sign  reversals 
Damping  ratio 

Amplitude-Distribution  Measures 
Mean,  median,  mode 
Standard  deviation,  variance,  range 
Minimum/maximum  value 
Root-mean- squared  error 
Absolute  average  error 

Frequency  Domain  Measures 


Power  spectral  density  function 
Bandwidth 
Peak  power 

Low/high  frequency  power 
Bode  plots,  fourler  coefficients 
Amplitude  ratio 
Phase  shift 

Transfer  function  model  parameters 
Quasl-1 Inear  describing  function 
Cross-over  model 


MEASURE  SELECTION 


The  present  state  of  the  art  requires  that  measurement  development 
must  start  with  a  good  analysis,  but  empirical  methods  are  required  to 
reduce  measures  to  a  small,  efficient  set. 

The  reduction  of  analytically  defined  measures  into  a  set  which  can 
be  shown  (operationally  and  statistically)  to  have  desired  properties  is 
called  the  measure  selection  process.  The  measure  selection  process  is 
based  on  multivariate  statistical  models  which  are  used  to  evaluate  the 
total  set  of  measures,  taken  together,  and  provide  measure  scaling 
(weighting)  Information. 

The  current  measure  selection  process  is  based  on  criteria  that  the 
final  measure  set  should  represent  a  comprehensive,  yet  minimum  set  of 
measures  trtvich:  1)  eliminate  redundant  forms  of  information;  2)  is 
sensitive  to  the  skill  changes  that  occur  during  training;  and  3)  has 
performance  prediction  qualities. 

The  first  step  In  the  development  of  measures  involves  specifying 
candidate  performance  measures  which,  in  the  judgment  of  the 
measurement  analyst,  might  contain  information  of  importance  for 
instructors,  adaptive  logics,  or  other  systems  which  are  to  manage 
training.  Examples  from  the  literature  can  be  used  along  with  common 
task  analytic  techniques  to  develop  analytic  measure  specifications. 

The  next  step  In  the  process  requires  collecting  empirical 
performance  data  during  training  to  provide  a  reliable  sample  for 
measure  selection  analyses.  Computer  measure  selection  techniques,  based 
on  multivariate  statistical  models,  are  used  to  reduce  the  candidate 
measures  to  a  final  measure  set  that  meets  statistical  and  operational 
criteria.  The  outcome  of  each  analysis  is  evaluated  and  merged 
(sometimes  subjectively)  into  a  reduced  set  of  measures  for  each  task. 

The  recommended  measure  sets  then  are  statistically  processed  to 
establish  weighting  coefficients  for  each  measure  (see  the  following 
section  of  this  appendix).  The  weights  reflect  the  relative  Importance 
of  each  measure  to  training,  scale  the  measures  on  a  common  basis,  and 
permit  suamation  of  the  many  measures  Into  one  score. 

MEASURE  SELECTION  TECHNIQUES 

Reduction  of  the  Initial  candidate  measure  set  to  something  that  is 
operationally  workable  should  be  based  on  an  analysis  of  data  collected 
through  a  trial  application  of  the  Initial  APM  system.  As  a  rather 
large  number  of  measures  is  typical  Initially,  a  number  of  pilots  and 
repetitions  Is  needed  for  an  adequate  statistical  sample.  The  criteria 
for  final  measure  set  selection  then  must  be  defined  In  quantitative, 
operational  form. 
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But,  on  what  ground  should  a  specific  measure  be  included  or 
excluded  for  further  consideration?  After  consideration  of  the  needs 
for  measurement  In  training  (Vreuls  and  Goldstein,  1974),  three  general 
criteria  are  recommended: 

1.  If  two  measures  provide  the  same  information  for  a  given 
application,  one  member  of  the  redundant  pair  may  be  discarded. 

2.  Measures  may  be  discarded  if  they  are  not  sensitive  to 
performance  differences  between  individuals.  The  measures 
retained  should  be  able  to  discriminate  between  "good"  and 
"poor"  performers,  and  performance  by  a  student  early  in 
training  compared  with  the  same  performance  later  In  training. 

3.  Measures  also  should  be  retained  if  they  lend  themselves  to 

early  prediction  of  operational  performance  to  be  achieved  by 
an  Individual;  for  example,  the  performance  level  to  be 

achieved  at  the  close  of  training,  or  the  prediction  of 
deficiencies  which  may  be  remedied  by  an  appropriate  change  In 
training. 

If  two  measures  correlate  highly,  then  one  of  the  pair  may  be 

removed  from  the  candidate  measure  set.  In  fact.  It  may  be  necessary  to 

remove  such  measures  for  the  proper  functioning  of  multivariate 

statistical  analyses  used  In  other  measurement  selection  criteria  (see 
the  following  sections  of  this  appendix).  However,  the  APM  system 
designer  also  must  ensure  that  small  differences  between  two  Imperfectly 
correlated  measures  are  not  Important;  for  example,  one  subject  of  a 
larger  group  may  be  sufficiently  different  that  the  measures  are 
definitely  uncorrelated,  but  only  for  him.  Further,  the  problem  of 

specifying  the  magnitude  Of  the  correlation  coefficient  (degree  of 
relationship)  for  determining  which  measures  will  be  considered 
redundant  remains  the  judgnent  of  the  APM  system  designer.  Further 
research  Is  required. 

A  multivariate  statistical  technique,  the  multiple  discriminant 
analysis.  Is  available  to  derive  a  discriminant  function  made  up  of  a 
composite  weighted  sum  of  measures  that  hold  potential  for  best 
differentiating  between  two  or  more  groups.  The  computed  weightings 
Indicate  the  relative  amounts  that  each  of  the  measures  contributes  to 
the  differentiation.  If  the  APM  system  designer  can  establish  test 
groups  which  are  known  to  be  different  in  terms  of  skill  levels,  and  If 
test  data  can  be  collected,  then  the  multiple  discriminant  analysis  can 
be  used  to  find  the  measures  within  the  candidate  measure  set  that 
maximize  the  differentiation.  The  weightings,  then,  are  key  to  the 
definition  of  measure  selection  criteria.  The  criteria  can  be  that  the 
measures  with  the  least  weights  are  discarded.  Of  course,  the  threshold 
level  for  measure  weights  now  must  be  left  to  the  APM  system  designer's 
judgment. 


Specific  details  of  the  APM  selection  criteria  used,  and  the 
statistics  of  the  selection  process,  are  best  presented  In  terms  of  the 
statistical  operations  needed,  which  are  sumnarlzed  In  the  following 
section. 

MEASURE  SELECTION  BY  DISCRIMINANT  ANALYSIS 

The  statistical  programs  generated  to  select  measurement  through 
discriminant  analyses  assume  that  a  battery  of  measures  has  been  taken 
for  each  group  of  subjects.  The  primary  purpose  of  these  statistical 
computer  programs  Is  to  Identify  the  measures  that  best  discriminate 
between  groups  of  known  skill  differences.  For  example,  a  pair  of 
groups  may  consist  of  experienced  and  inexperienced  pilots.  The 
statistical  procedure  eliminates  measures  that  do  not  contribute  to  the 
ability  to  tell  the  groups  apart. 

Computer  programs  for  measure  selection  based  on  discriminant 
analysis  Iteratively  discard  candidate  measures  until  a  minimum  set  of 
statistically  defined  measures  is  selected  by  the  APM  analyst.  The 
iterative  process  stops  when  one  of  two  criteria  Is  met:  1)  the  total 
number  of  remaining  measures  Is  less  than  the  minimum  number  of  factors 
determined  through  a  principal  components  analysis;  or  2)  discarding 
another  measure  will  reduce  discrimination  to  an  unacceptable  level. 

The  above  procedure  Is  satisfactory  unless  some  of  the  measures  are 
highly  correlated;  then  the  ability  of  the  measure  set  to  differentiate 
between  groups  cannot  be  clearly  attributed  to  either  of  a  pair  of 
correlating  measures.  The  recommended  procedure,  therefore,  first 
performs  a  correlation  analysis,  and  one  of  the  pair  of  highly 
correlative  measures  must  be  discarded.  The  procedure  then  Is  continued 
(with  selected  discards)  until  the  measurement  analyst  Is  satisfied. 
Obviously,  further  research  Is  needed  so  that  automated  measures 
ultimately  can  be  determined  without  excessive  reliance  on  this  complex 
developmental  procedure. 

MEASURE  SELECTION  BY  CANONICAL  CORRELATION  ANALYSIS 

Another  series  of  data  analysis  programs  is  designed  to  select 
measures  that  relate  performance  demonstrated  at  one  time  In  training  to 
that  at  another  time.  The  basis  of  the  method  Is  a  canonical 
correlation  analysis  which  derives  a  linear  combination  of  the  measures, 
and  maximizes  the  correlation  between  the  linear  combination  of  one  set 
of  measures  in  relation  to  another  set  of  measures.  If  the  following 
linear  combinations  are  formed: 


Y1  «  a2x2+ 
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where  x  and  z  are  the  same  measures  collected  at  different  points  in  the 
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training  sequence,  then  canonical  correlation  analysis  determines  the 
coefficients  a  and’b  so  that  Y1  and  Y2  are  maximally  correlated. 


The  quantities  Y1  and  Y2  are  factors  of  their  respective  data 
groups.  Computer  programs  generate  the  factor  structure  for  each  set  of 
data  Mhich  displays  the  correlation  between  each  measure  and  factor. 
The  factor  which  correlates  between  groups  best  is  also  indicated,  and 
It  is  this  factor  which  Is  used  for  measure  selection.  The  measure 
which  correlates  least  with  this  factor  contributes  least  to  between 
group  correlation. 

MEASURE  SET  SELECTION 

The  objectives  of  the  previous  measure  selection  processes  are  to 
define  the  set  of  measures  that  best  discriminated  among  skill  levels 
and  predict  performance  for  each  task  and  operational  variation.  The 
results  of  the  discriminant  analysis  and  canonical  correlation  analyses 
are  brought  together  to  find  commonalities  and  differences  in  order  to 
recommend  measurement  which  will  contain  both  discriminative  and 
predictive  information.  Also,  it  Is  necessary  to  determine  the 
weighting  coefficients  for  each  measure  to  reflect  the  training 
importance  of  each  measure  and  to  scale  each  measure  on  a  common  basis 
for  possible  summation  Into  an  overall  performance  score. 

Final  measure  set  selection  remains  a  judgmental  procedure  based  on 
experience  and  objective  data.  If  one  wishes  to  use  these  criteria, 
minimum  measure  sets  which  result  from  the  discriminant  analysis  can  be 
entered  Into  a  student  performance  data  table  for  each  maneuver  segment 
and  task  variation,  such  as  aircraft  weight  change  or  addition  of 
turbulent  air.  To  each  table  are  added  any  measures  produced  by  the 
canonical  process. 

Resulting  measure  sets  are  examined  to  ensure  that  all  aircraft 
outer  loops  representing  task  Instruction  are  present.  Measures  that 
appeared  infrequently  (once  or  twice  for  any  maneuver  segment)  may  be 
eliminated  If  the  Information  they  contain  Is  represented  by  a  high 
correlation  (.70  or  greater)  with  another  comparable  measure  that  is 
contained  in  the  set,  and/or  they  have  been  added  to  the  measurement 
table  by  the  canonical  process  and  are  weakly  related  (loadings  less 
than  .20)  to  the  canonical  factors. 

Finally,  discriminant  analysis  can  be  modified  to  perform  an 
analysis  on  the  recoomended  measure  set.  The  overall  analysis  assures 
that  a  significant  discriminant  function  Is  retained.  Also,  the  weights 
assigned  to  each  measure  of  the  final  set  are  computed.  The  weights 
provide  scaling  Information  for  each  measure  relative  to  other  measures 
In  the  set  and  coefficients  for  the  linear  sumnatlon  of  the  measures 
Into  a  single  score. 
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APPENDIX  C 
GLOSSARY  OF  TERMS 
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AAT  -  Automated  adaptive  training,  which  Is  a  complex  and  evolving 
technology  In  which  Instructional  objectives  and  strategies  that 
are  Implemented  as  computer  software  are  used  to  automatically 
tailor  (adapt)  Instruction  to  Individual  student  learning  needs. 

ATO  -  Aircrew  training  device.  These  training  media  include  cockpit 
familiarization  and  procedures  trainers,  operational  flight 
trainers,  part-task  trainers,  weapon  system  trainers  and  full 
mission  trainers. 

AUGMENTATION  -  Providing  Information  which  does  not  exist  in  the  real 
world,  or  an  enhancement  of  naturally  occurring  information. 

AUTO-DEMO  -  Automated  Demonstrations.  An  instructional  feature  that 

permits  the  standardized  presentation  of  a  mission  segment  or 
entire  simulated  flight.  All  cues  of  consequence,  including  the 
visual  scene  (if  present),  motion  system,  primary  flight  controls 
and  displays,  crew  communications  and  sensor  displays,  are 
reproduced  for  the  selected  mission  segment  or  maneuver  through 
computer  control. 

AUTOMATED  COACHING  -  Providing  automatically  guidance  messages  to 
students  that  are  designed  to  assist  the  student  in  correcting  a 
performance  deficiency. 

AUTOMATED  CONTROLLERS  -  Computer-based  systems  that  use  mathematical 
models  to  determine  controller  messages  and  issue  controller 
messages  using  computer  generated  speech. 

AUTOMATED  CUING  -  The  automated  capability  to  provide  performance 
alerting  messages  to  students  when  their  performance  does  not  meet 
established  standards. 

AUTOMATED  PERFORMANCE  ALERTS  -  Signals  intended  to  enhance  the 
monitoring  of  student  performance  by  combining  the  capabilities  of 
automated  performance  measurement  with  automated  alerting  signal 
generation  for  Instructors  and/or  students. 

AUTOMATED  PERFORMANCE  MEASUREMENT  (APM)  -  The  computer-based  application 
of  technology  to  monitoring,  recording,  processing  *  and  displaying 
objective,  quantitative  information  that  describes  student 
performance  and  assists  In  diagnosing  student  learning  problems. 

CCT  -  Combat  crew  training. 

CCTV  -  Closed  circuit  television. 

CPT  -  Cockpit  procedure*  trainer. 

CT  -  Continuation  training:  training  conducted  routinely  In  operational 
squadrons,  or  proficiency  training  conducted  periodically. 


CUE  -  In  this  report,  cue  means  some  critical  feature  which  gives 
important  information  to  a  pilot  or  other  aircrew  member.  There  is 
no  commonly  accepted  definition  of  cue  as  used  with  respect  to 
fidelity. 

DIAGNOSE  -  To  determine  the  reasons  underlying  less  than  acceptable 
performance. 

FIDELITY  -  The  extent  to  which  cue  and  response  capabilities  in  an  ATD 
allow  for  the  learning  and  practice  of  specific  tasks  so  that  what 
is  learned  will  enhance  performance  of  the  tasks  in  the  operational 
environment.  Also  see:  physical  fidelity;  task  fidelity;  and 
realism. 

FIELD  OF  VIEW  -  The  dimensions  of  the  area  of  a  visual  display  which  can 
be  seen. 

FLIGHT  TRAINING  MODEL  -  Patterning  the  way  aircrew  training  is  done  in 
an  ATD  directly  after  the  way  comparable  training  is  done  in  the 
aircraft;  needlessly  imposing  limitations  on  ATD  use  that  stem  from 
the  ways  aircraft  must  be  used  for  training. 

FREEZE  -  An  instructional  support  feature  that  allows  simulator 
parameters  to  be  fixed  at  values  existing  when  freeze  is  operated. 

IMAGE  -  The  picture  or  scene  created  by  a  simulator  visual  system  which 
is  viewed  by  a  pilot  or  other  aircrew  member. 

IMAGE  QUALITY  -  Characteristics  of  the  appearance  of  an  image. 
Independent  of  the  scene  content  of  the  image. 

INITIALIZATION  -  Initial ization  Involves  specifying,  usually  from  the 
Instructor/operator  console,  the  parameters  of  Interest  and  their 
values  for  repositioning  and  reconfiguring  an  ATD  through  computer 
control . 

INSTRUCTIONAL  SUPPORT  FEATURE  -  ATD  hardware  and  software  capabilities 
that  allow  the  Instructor  to  manipulate,  supplement  and  otherwise 
control  the  student's  learning  experiences  with  the  Intent  of 
promoting  the  rate  at  which  skills  are  learned  and  maximizing  the 
levels  of  skills  achieved. 

IRON  PILOTS  -  Computer  controlled  adversaries,  typically  used  In  air 
combat  maneuvering  training  Iti  ATDs. 

ISO  -  Instructional  system  development:  procedural  approaches  to  the 
analysis  of  training  requirements  and  the  development  of  training 
systems. 
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MULTI-VARIATE  -  A  number  of  variables  that  are  combined  to  provide  an 
indicator  of  performance. 

OFT  -  Operational  flight  trainer. 

PHYSICAL  FIDELITY  -  The  degree  of  structural  or  dynamic  correspondence 
of  an  ATO  to  the  aircraft  it  represents. 

PRACTICE  -  Deliberate  participation  In  activities  for  the  purpose  of 
learning  or  mastering  skills  that  depend  on  the  thoughts  and  motor 
actions  involved  in  the  activities. 

PROGRAMMED  MISSION  SCENARIOS  -  Highly  structured  sets  of  events  that  are 
caused  to  occur  automatically,  under  computer  control. 

RECORD  AND  REPLAY  -  The  instructional  feature  that  provides  the 
capability  in  an  ATD  to  record  relevant  system  parameters  and  then 
use  these  data  to  present  student  pilot  performance  in  a  review 
mode  in  the  ATD. 

REALISM  -  The  extent  to  which  an  aircrew  member's  experiences  in  an  ATD 
correspond  to  experiences  as  they  actually  would  occur  in  an 
aircraft  under  a  given  set  of  conditions.  Also  see  physical 
fidelity. 

RESOLUTION  -  The  smallest  separation  between  two  objects  in  a  display 
which  can  be  detected,  usually  by  the  human  eye. 

RESPONSE  -  Any  motor,  perceptual  or  mental  act  by  a  person;  generally 
refers  to  an  element  of  an  overall  action  as  opposed  to  the  overall 
action  itself. 

RETENTION  -  The  capacity  to  remember  task  requirements  and  perform 
accordingly  after  a  lapse  of  time  during  which  the  task  has  not 
been  practiced. 

STRES  -  Simulator  Training  Requirements  and  Effectiveness  Study. 

TASK  FIDELITY  -  The  degree  of  correspondence  of  cues  and  responses 
accompanying  task  performance  in  an  ATD  to  those  characteristics  of 
analogous  performance  in  an  aircraft. 

TRAINING  EFFECTIVENESS  -  The  training  benefit  gained  in  terms  of 
operational  readiness.  Also,  the  thoroughness  with  which  training 
objectives  have  been  achieved,  regardless  of  training  efficiency. 

TRAINING  EFFICIENCY  -  The  extent  to  which  training  resources  (Including 
time)  are  used  economically  while  achieving  training  effectiveness. 
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TRAINING  OBJECTIVES  -  Precise  statements  of  the  goals  of  training  which 
set  forth  the  tasks  to  be  performed,  the  performance  standards  to 
be  met  for  each  task,  and  the  conditions  under  which  task 
performance  is  to  be  demonstrated. 

TRAINING  REQUIREMENTS  -  General  statements  of  job  performance  skills 
required  for  operational  proficiency.  Also,  general  statements  of 
Job  performance  skills  that  require  periodic  practice  in  order  to 
maintain  proficiency. 

TRANSFER  OF  TRAINING  -  The  use  cf  skills  learned  in  one  context  (e.g., 
an  ATD)  in  a  substantially  different  context  (e.g.,  an  aircraft). 
The  carry-forward  of  trained  performance  to  real  world 
applications. 

TRANSITION  TRAINING  -  Training  for  aircrew  members  transitioning  to 
different  operational  aircraft. 

UPT  -  Undergraduate  pilot  training:  initial  pilot  qualification 
training. 

WST  -  Weapon  system  trainer. 


